[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251023 02:03:10.882951 20804 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.81.62:35173
I20251023 02:03:10.884500 20804 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 02:03:10.885526 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 02:03:10.903353 20804 server_base.cc:1047] running on GCE node
W20251023 02:03:10.902349 20811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:10.902388 20813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:10.903647 20810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:12.074685 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251023 02:03:12.074937 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:03:12.075148 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761184992075121 us; error 0 us; skew 500 ppm
I20251023 02:03:12.075986 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:03:12.086370 20804 webserver.cc:492] Webserver started at http://127.20.81.62:40381/ using document root <none> and password file <none>
I20251023 02:03:12.087684 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:03:12.087982 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:03:12.088614 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:03:12.095923 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/master-0-root/instance:
uuid: "c82c7a6ce1a14fe897847751021e80b6"
format_stamp: "Formatted at 2025-10-23 02:03:12 on dist-test-slave-6q7l"
I20251023 02:03:12.103777 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20251023 02:03:12.108896 20819 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:12.110738 20804 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251023 02:03:12.111166 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/master-0-root
uuid: "c82c7a6ce1a14fe897847751021e80b6"
format_stamp: "Formatted at 2025-10-23 02:03:12 on dist-test-slave-6q7l"
I20251023 02:03:12.111536 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:03:12.185457 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:03:12.187047 20804 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 02:03:12.187597 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:03:12.260660 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.62:35173
I20251023 02:03:12.260746 20870 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.62:35173 every 8 connection(s)
I20251023 02:03:12.268636 20871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:12.290798 20871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6: Bootstrap starting.
I20251023 02:03:12.296715 20871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:12.298593 20871 log.cc:826] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6: Log is configured to *not* fsync() on all Append() calls
I20251023 02:03:12.303077 20871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6: No bootstrap required, opened a new log
I20251023 02:03:12.324882 20871 raft_consensus.cc:359] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82c7a6ce1a14fe897847751021e80b6" member_type: VOTER }
I20251023 02:03:12.325714 20871 raft_consensus.cc:385] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:12.326092 20871 raft_consensus.cc:740] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82c7a6ce1a14fe897847751021e80b6, State: Initialized, Role: FOLLOWER
I20251023 02:03:12.327203 20871 consensus_queue.cc:260] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82c7a6ce1a14fe897847751021e80b6" member_type: VOTER }
I20251023 02:03:12.327821 20871 raft_consensus.cc:399] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 02:03:12.328061 20871 raft_consensus.cc:493] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 02:03:12.328363 20871 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:12.332435 20871 raft_consensus.cc:515] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82c7a6ce1a14fe897847751021e80b6" member_type: VOTER }
I20251023 02:03:12.333108 20871 leader_election.cc:304] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c82c7a6ce1a14fe897847751021e80b6; no voters:
I20251023 02:03:12.334743 20871 leader_election.cc:290] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 02:03:12.335039 20874 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:12.337141 20874 raft_consensus.cc:697] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [term 1 LEADER]: Becoming Leader. State: Replica: c82c7a6ce1a14fe897847751021e80b6, State: Running, Role: LEADER
I20251023 02:03:12.338146 20874 consensus_queue.cc:237] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82c7a6ce1a14fe897847751021e80b6" member_type: VOTER }
I20251023 02:03:12.339246 20871 sys_catalog.cc:565] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 02:03:12.349045 20875 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c82c7a6ce1a14fe897847751021e80b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82c7a6ce1a14fe897847751021e80b6" member_type: VOTER } }
I20251023 02:03:12.348918 20876 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c82c7a6ce1a14fe897847751021e80b6. Latest consensus state: current_term: 1 leader_uuid: "c82c7a6ce1a14fe897847751021e80b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82c7a6ce1a14fe897847751021e80b6" member_type: VOTER } }
I20251023 02:03:12.349675 20875 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [sys.catalog]: This master's current role is: LEADER
I20251023 02:03:12.349725 20876 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [sys.catalog]: This master's current role is: LEADER
I20251023 02:03:12.353952 20883 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 02:03:12.364326 20883 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 02:03:12.366139 20804 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251023 02:03:12.379992 20883 catalog_manager.cc:1357] Generated new cluster ID: c51da4492d3f466fb089c77598646d7e
I20251023 02:03:12.380297 20883 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 02:03:12.401168 20883 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 02:03:12.403332 20883 catalog_manager.cc:1514] Loading token signing keys...
I20251023 02:03:12.418049 20883 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6: Generated new TSK 0
I20251023 02:03:12.419035 20883 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 02:03:12.432823 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:03:12.441663 20893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:12.443176 20894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:12.447566 20896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:12.448221 20804 server_base.cc:1047] running on GCE node
I20251023 02:03:12.448988 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251023 02:03:12.449203 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:03:12.449358 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761184992449341 us; error 0 us; skew 500 ppm
I20251023 02:03:12.449965 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:03:12.454296 20804 webserver.cc:492] Webserver started at http://127.20.81.1:41495/ using document root <none> and password file <none>
I20251023 02:03:12.454844 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:03:12.455075 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:03:12.455396 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:03:12.456554 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-0-root/instance:
uuid: "0130a1960e0740469b5291134a4e78a3"
format_stamp: "Formatted at 2025-10-23 02:03:12 on dist-test-slave-6q7l"
I20251023 02:03:12.462352 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20251023 02:03:12.466593 20901 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:12.467514 20804 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20251023 02:03:12.467798 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-0-root
uuid: "0130a1960e0740469b5291134a4e78a3"
format_stamp: "Formatted at 2025-10-23 02:03:12 on dist-test-slave-6q7l"
I20251023 02:03:12.468097 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:03:12.484093 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:03:12.485405 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:03:12.487699 20804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 02:03:12.492224 20804 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 02:03:12.492476 20804 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:12.492729 20804 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 02:03:12.492902 20804 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:12.588613 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.1:35337
I20251023 02:03:12.588686 20963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.1:35337 every 8 connection(s)
I20251023 02:03:12.595422 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:03:12.606336 20968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:12.609176 20969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:12.619352 20804 server_base.cc:1047] running on GCE node
W20251023 02:03:12.621063 20971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:12.622319 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20251023 02:03:12.622398 20964 heartbeater.cc:344] Connected to a master server at 127.20.81.62:35173
W20251023 02:03:12.622582 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:03:12.622800 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761184992622784 us; error 0 us; skew 500 ppm
I20251023 02:03:12.622920 20964 heartbeater.cc:461] Registering TS with master...
I20251023 02:03:12.623440 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:03:12.624058 20964 heartbeater.cc:507] Master 127.20.81.62:35173 requested a full tablet report, sending...
I20251023 02:03:12.627266 20804 webserver.cc:492] Webserver started at http://127.20.81.2:41975/ using document root <none> and password file <none>
I20251023 02:03:12.627807 20836 ts_manager.cc:194] Registered new tserver with Master: 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337)
I20251023 02:03:12.628012 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:03:12.628278 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:03:12.628620 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:03:12.630141 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-1-root/instance:
uuid: "30052fa1e09b406e97bd508d13cac415"
format_stamp: "Formatted at 2025-10-23 02:03:12 on dist-test-slave-6q7l"
I20251023 02:03:12.630784 20836 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41706
I20251023 02:03:12.636222 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251023 02:03:12.640275 20976 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:12.641495 20804 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20251023 02:03:12.641811 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-1-root
uuid: "30052fa1e09b406e97bd508d13cac415"
format_stamp: "Formatted at 2025-10-23 02:03:12 on dist-test-slave-6q7l"
I20251023 02:03:12.642097 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:03:12.664827 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:03:12.666184 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:03:12.668407 20804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 02:03:12.671325 20804 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 02:03:12.671533 20804 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:12.671765 20804 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 02:03:12.671918 20804 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:12.764983 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.2:44407
I20251023 02:03:12.765097 21038 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.2:44407 every 8 connection(s)
I20251023 02:03:12.773301 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:03:12.784960 21043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:12.785291 21042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:12.788166 21039 heartbeater.cc:344] Connected to a master server at 127.20.81.62:35173
I20251023 02:03:12.788656 21039 heartbeater.cc:461] Registering TS with master...
I20251023 02:03:12.789903 21039 heartbeater.cc:507] Master 127.20.81.62:35173 requested a full tablet report, sending...
I20251023 02:03:12.792891 20836 ts_manager.cc:194] Registered new tserver with Master: 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:12.795050 20836 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41714
I20251023 02:03:12.796785 20804 server_base.cc:1047] running on GCE node
W20251023 02:03:12.797447 21045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:12.798619 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251023 02:03:12.798884 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:03:12.799088 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761184992799070 us; error 0 us; skew 500 ppm
I20251023 02:03:12.799786 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:03:12.802613 20804 webserver.cc:492] Webserver started at http://127.20.81.3:43231/ using document root <none> and password file <none>
I20251023 02:03:12.803232 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:03:12.803478 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:03:12.803786 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:03:12.805303 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-2-root/instance:
uuid: "1796d6cb245b41a29d0b808df3338012"
format_stamp: "Formatted at 2025-10-23 02:03:12 on dist-test-slave-6q7l"
I20251023 02:03:12.811034 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251023 02:03:12.815152 21050 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:12.815968 20804 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251023 02:03:12.816304 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-2-root
uuid: "1796d6cb245b41a29d0b808df3338012"
format_stamp: "Formatted at 2025-10-23 02:03:12 on dist-test-slave-6q7l"
I20251023 02:03:12.816648 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761184990718903-20804-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:03:12.856516 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:03:12.858242 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:03:12.859917 20804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 02:03:12.862778 20804 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 02:03:12.863055 20804 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:12.863343 20804 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 02:03:12.863551 20804 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:12.947248 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.3:33625
I20251023 02:03:12.947688 21112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.3:33625 every 8 connection(s)
I20251023 02:03:12.963689 21113 heartbeater.cc:344] Connected to a master server at 127.20.81.62:35173
I20251023 02:03:12.964105 21113 heartbeater.cc:461] Registering TS with master...
I20251023 02:03:12.964859 21113 heartbeater.cc:507] Master 127.20.81.62:35173 requested a full tablet report, sending...
I20251023 02:03:12.966831 20836 ts_manager.cc:194] Registered new tserver with Master: 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:12.967496 20804 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014910485s
I20251023 02:03:12.968393 20836 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41722
I20251023 02:03:12.971321 20804 test_util.cc:276] Using random seed: -706491874
I20251023 02:03:13.015246 20836 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41724:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 02:03:13.018847 20836 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 02:03:13.115777 20929 tablet_service.cc:1505] Processing CreateTablet for tablet 69c64628ec8443168d18dbd1f66cc92e (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251023 02:03:13.121961 20926 tablet_service.cc:1505] Processing CreateTablet for tablet 49e9de91bc8041e98d22df0219352ba7 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251023 02:03:13.121374 20927 tablet_service.cc:1505] Processing CreateTablet for tablet 648a6aac62494cdda1eb0ff98736376b (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251023 02:03:13.124522 20928 tablet_service.cc:1505] Processing CreateTablet for tablet 676f88b0f9984cba93d7407ac7d88bd3 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251023 02:03:13.122785 20925 tablet_service.cc:1505] Processing CreateTablet for tablet 6c36810008544d7fa5c781cae7b1d884 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251023 02:03:13.126381 20929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c64628ec8443168d18dbd1f66cc92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.125996 20924 tablet_service.cc:1505] Processing CreateTablet for tablet fb614549d596431bbedda15b128caf3e (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251023 02:03:13.128355 20923 tablet_service.cc:1505] Processing CreateTablet for tablet be886e390b59440393859aef4d6e61f5 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251023 02:03:13.129230 20927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648a6aac62494cdda1eb0ff98736376b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.134703 20926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49e9de91bc8041e98d22df0219352ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.135506 20924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb614549d596431bbedda15b128caf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.137033 20925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c36810008544d7fa5c781cae7b1d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.137708 20923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be886e390b59440393859aef4d6e61f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.138543 20928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 676f88b0f9984cba93d7407ac7d88bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.141901 20922 tablet_service.cc:1505] Processing CreateTablet for tablet 6609dd61840640d796684211f3164998 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251023 02:03:13.145310 20922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6609dd61840640d796684211f3164998. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.173926 21004 tablet_service.cc:1505] Processing CreateTablet for tablet 69c64628ec8443168d18dbd1f66cc92e (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251023 02:03:13.175375 21004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c64628ec8443168d18dbd1f66cc92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.193495 21003 tablet_service.cc:1505] Processing CreateTablet for tablet 676f88b0f9984cba93d7407ac7d88bd3 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251023 02:03:13.195955 20997 tablet_service.cc:1505] Processing CreateTablet for tablet 6609dd61840640d796684211f3164998 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251023 02:03:13.191787 20998 tablet_service.cc:1505] Processing CreateTablet for tablet be886e390b59440393859aef4d6e61f5 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251023 02:03:13.194298 20999 tablet_service.cc:1505] Processing CreateTablet for tablet fb614549d596431bbedda15b128caf3e (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251023 02:03:13.201448 21002 tablet_service.cc:1505] Processing CreateTablet for tablet 648a6aac62494cdda1eb0ff98736376b (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251023 02:03:13.190205 21000 tablet_service.cc:1505] Processing CreateTablet for tablet 6c36810008544d7fa5c781cae7b1d884 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251023 02:03:13.191432 21001 tablet_service.cc:1505] Processing CreateTablet for tablet 49e9de91bc8041e98d22df0219352ba7 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251023 02:03:13.206838 20996 tablet_service.cc:1505] Processing CreateTablet for tablet d54367d776794a079ca3bf28b8194efc (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251023 02:03:13.198827 20921 tablet_service.cc:1505] Processing CreateTablet for tablet d54367d776794a079ca3bf28b8194efc (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251023 02:03:13.210620 20921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d54367d776794a079ca3bf28b8194efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.213965 20997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6609dd61840640d796684211f3164998. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.222455 21003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 676f88b0f9984cba93d7407ac7d88bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.227993 20998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be886e390b59440393859aef4d6e61f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.231526 20996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d54367d776794a079ca3bf28b8194efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.233207 21002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648a6aac62494cdda1eb0ff98736376b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.235411 21001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49e9de91bc8041e98d22df0219352ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.239023 20999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb614549d596431bbedda15b128caf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.242420 21000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c36810008544d7fa5c781cae7b1d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.267236 21078 tablet_service.cc:1505] Processing CreateTablet for tablet 69c64628ec8443168d18dbd1f66cc92e (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251023 02:03:13.269658 21078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c64628ec8443168d18dbd1f66cc92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.296459 21131 tablet_bootstrap.cc:492] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3: Bootstrap starting.
I20251023 02:03:13.288935 21077 tablet_service.cc:1505] Processing CreateTablet for tablet 676f88b0f9984cba93d7407ac7d88bd3 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251023 02:03:13.326733 21077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 676f88b0f9984cba93d7407ac7d88bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.333287 21076 tablet_service.cc:1505] Processing CreateTablet for tablet 648a6aac62494cdda1eb0ff98736376b (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251023 02:03:13.336447 21076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648a6aac62494cdda1eb0ff98736376b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.346271 21075 tablet_service.cc:1505] Processing CreateTablet for tablet 49e9de91bc8041e98d22df0219352ba7 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251023 02:03:13.347667 21075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49e9de91bc8041e98d22df0219352ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.352972 21131 tablet_bootstrap.cc:654] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.359269 21073 tablet_service.cc:1505] Processing CreateTablet for tablet fb614549d596431bbedda15b128caf3e (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251023 02:03:13.362221 21073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb614549d596431bbedda15b128caf3e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 02:03:13.362123 20884 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251023 02:03:13.356652 21074 tablet_service.cc:1505] Processing CreateTablet for tablet 6c36810008544d7fa5c781cae7b1d884 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251023 02:03:13.372781 21074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c36810008544d7fa5c781cae7b1d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.381537 21072 tablet_service.cc:1505] Processing CreateTablet for tablet be886e390b59440393859aef4d6e61f5 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251023 02:03:13.385129 21072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be886e390b59440393859aef4d6e61f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.393010 21071 tablet_service.cc:1505] Processing CreateTablet for tablet 6609dd61840640d796684211f3164998 (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251023 02:03:13.395051 21071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6609dd61840640d796684211f3164998. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.416270 21070 tablet_service.cc:1505] Processing CreateTablet for tablet d54367d776794a079ca3bf28b8194efc (DEFAULT_TABLE table=test-workload [id=6a701a3755b141ed82a6b28710de369b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251023 02:03:13.422995 21070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d54367d776794a079ca3bf28b8194efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:13.432863 21133 tablet_bootstrap.cc:492] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415: Bootstrap starting.
I20251023 02:03:13.463029 21131 tablet_bootstrap.cc:492] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3: No bootstrap required, opened a new log
I20251023 02:03:13.470774 21131 ts_tablet_manager.cc:1403] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3: Time spent bootstrapping tablet: real 0.208s user 0.009s sys 0.019s
I20251023 02:03:13.486126 21134 tablet_bootstrap.cc:492] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012: Bootstrap starting.
I20251023 02:03:13.497215 21133 tablet_bootstrap.cc:654] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.497838 21134 tablet_bootstrap.cc:654] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.495740 21131 raft_consensus.cc:359] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.499384 21131 raft_consensus.cc:385] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.499715 21131 raft_consensus.cc:740] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.500444 21131 consensus_queue.cc:260] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.517279 21131 ts_tablet_manager.cc:1434] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3: Time spent starting tablet: real 0.046s user 0.011s sys 0.007s
I20251023 02:03:13.518157 20964 heartbeater.cc:499] Master 127.20.81.62:35173 was elected leader, sending a full tablet report...
I20251023 02:03:13.518442 21131 tablet_bootstrap.cc:492] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3: Bootstrap starting.
I20251023 02:03:13.523653 21133 tablet_bootstrap.cc:492] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415: No bootstrap required, opened a new log
I20251023 02:03:13.523808 21134 tablet_bootstrap.cc:492] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012: No bootstrap required, opened a new log
I20251023 02:03:13.524122 21133 ts_tablet_manager.cc:1403] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415: Time spent bootstrapping tablet: real 0.092s user 0.005s sys 0.015s
I20251023 02:03:13.524235 21134 ts_tablet_manager.cc:1403] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.009s
I20251023 02:03:13.527482 21134 raft_consensus.cc:359] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.527784 21133 raft_consensus.cc:359] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.528331 21134 raft_consensus.cc:385] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.528519 21133 raft_consensus.cc:385] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.528877 21134 raft_consensus.cc:740] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.528896 21133 raft_consensus.cc:740] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.529891 21134 consensus_queue.cc:260] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.530443 21133 consensus_queue.cc:260] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.545693 21131 tablet_bootstrap.cc:654] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.556051 21113 heartbeater.cc:499] Master 127.20.81.62:35173 was elected leader, sending a full tablet report...
I20251023 02:03:13.556799 21134 ts_tablet_manager.cc:1434] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012: Time spent starting tablet: real 0.032s user 0.011s sys 0.020s
I20251023 02:03:13.557504 21133 ts_tablet_manager.cc:1434] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415: Time spent starting tablet: real 0.033s user 0.019s sys 0.011s
I20251023 02:03:13.557938 21134 tablet_bootstrap.cc:492] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012: Bootstrap starting.
I20251023 02:03:13.564620 21134 tablet_bootstrap.cc:654] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.567058 21039 heartbeater.cc:499] Master 127.20.81.62:35173 was elected leader, sending a full tablet report...
I20251023 02:03:13.567498 21133 tablet_bootstrap.cc:492] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415: Bootstrap starting.
I20251023 02:03:13.576459 21133 tablet_bootstrap.cc:654] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.577656 21134 tablet_bootstrap.cc:492] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012: No bootstrap required, opened a new log
I20251023 02:03:13.578171 21134 ts_tablet_manager.cc:1403] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.006s
I20251023 02:03:13.581027 21134 raft_consensus.cc:359] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.582008 21134 raft_consensus.cc:385] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.582369 21134 raft_consensus.cc:740] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.583164 21134 consensus_queue.cc:260] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.585485 21134 ts_tablet_manager.cc:1434] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251023 02:03:13.586525 21134 tablet_bootstrap.cc:492] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012: Bootstrap starting.
I20251023 02:03:13.593099 21134 tablet_bootstrap.cc:654] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012: Neither blocks nor log segments found. Creating new log.
W20251023 02:03:13.596683 20965 tablet.cc:2378] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 02:03:13.603462 21131 tablet_bootstrap.cc:492] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3: No bootstrap required, opened a new log
I20251023 02:03:13.603943 21131 ts_tablet_manager.cc:1403] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3: Time spent bootstrapping tablet: real 0.086s user 0.018s sys 0.011s
I20251023 02:03:13.604656 21134 tablet_bootstrap.cc:492] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012: No bootstrap required, opened a new log
I20251023 02:03:13.605155 21134 ts_tablet_manager.cc:1403] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20251023 02:03:13.607388 21131 raft_consensus.cc:359] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.608129 21131 raft_consensus.cc:385] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.608508 21131 raft_consensus.cc:740] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.608094 21134 raft_consensus.cc:359] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.609104 21134 raft_consensus.cc:385] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.609426 21134 raft_consensus.cc:740] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.609227 21131 consensus_queue.cc:260] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.610142 21134 consensus_queue.cc:260] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.612694 21134 ts_tablet_manager.cc:1434] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20251023 02:03:13.613718 21134 tablet_bootstrap.cc:492] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012: Bootstrap starting.
I20251023 02:03:13.618697 21133 tablet_bootstrap.cc:492] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415: No bootstrap required, opened a new log
I20251023 02:03:13.620242 21134 tablet_bootstrap.cc:654] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.620587 21131 ts_tablet_manager.cc:1434] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
I20251023 02:03:13.621150 21133 ts_tablet_manager.cc:1403] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415: Time spent bootstrapping tablet: real 0.054s user 0.017s sys 0.004s
I20251023 02:03:13.623917 21133 raft_consensus.cc:359] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.624617 21133 raft_consensus.cc:385] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.624918 21133 raft_consensus.cc:740] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.625646 21133 consensus_queue.cc:260] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.631426 21131 tablet_bootstrap.cc:492] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3: Bootstrap starting.
I20251023 02:03:13.632090 21133 ts_tablet_manager.cc:1434] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20251023 02:03:13.633112 21133 tablet_bootstrap.cc:492] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415: Bootstrap starting.
I20251023 02:03:13.634534 21134 tablet_bootstrap.cc:492] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012: No bootstrap required, opened a new log
I20251023 02:03:13.635125 21134 ts_tablet_manager.cc:1403] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.012s
I20251023 02:03:13.638026 21134 raft_consensus.cc:359] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.638872 21131 tablet_bootstrap.cc:654] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.638933 21134 raft_consensus.cc:385] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.639340 21133 tablet_bootstrap.cc:654] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.639423 21134 raft_consensus.cc:740] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.640584 21134 consensus_queue.cc:260] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.643970 21134 ts_tablet_manager.cc:1434] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251023 02:03:13.645002 21134 tablet_bootstrap.cc:492] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012: Bootstrap starting.
I20251023 02:03:13.652196 21134 tablet_bootstrap.cc:654] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.662177 21131 tablet_bootstrap.cc:492] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3: No bootstrap required, opened a new log
I20251023 02:03:13.662699 21131 ts_tablet_manager.cc:1403] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.007s
I20251023 02:03:13.664139 21134 tablet_bootstrap.cc:492] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012: No bootstrap required, opened a new log
I20251023 02:03:13.664657 21134 ts_tablet_manager.cc:1403] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20251023 02:03:13.665531 21131 raft_consensus.cc:359] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.666343 21131 raft_consensus.cc:385] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.666662 21131 raft_consensus.cc:740] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.667649 21131 consensus_queue.cc:260] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.667934 21134 raft_consensus.cc:359] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.668509 21133 tablet_bootstrap.cc:492] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415: No bootstrap required, opened a new log
I20251023 02:03:13.668886 21134 raft_consensus.cc:385] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.669067 21133 ts_tablet_manager.cc:1403] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.011s
I20251023 02:03:13.669431 21134 raft_consensus.cc:740] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.670435 21134 consensus_queue.cc:260] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.671911 21133 raft_consensus.cc:359] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.672616 21133 raft_consensus.cc:385] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.672976 21133 raft_consensus.cc:740] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.673301 21134 ts_tablet_manager.cc:1434] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20251023 02:03:13.673748 21133 consensus_queue.cc:260] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.674369 21134 tablet_bootstrap.cc:492] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012: Bootstrap starting.
I20251023 02:03:13.676239 21133 ts_tablet_manager.cc:1434] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20251023 02:03:13.677141 21133 tablet_bootstrap.cc:492] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415: Bootstrap starting.
I20251023 02:03:13.681249 21134 tablet_bootstrap.cc:654] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.683370 21133 tablet_bootstrap.cc:654] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.687106 21131 ts_tablet_manager.cc:1434] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3: Time spent starting tablet: real 0.024s user 0.006s sys 0.001s
I20251023 02:03:13.688196 21131 tablet_bootstrap.cc:492] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3: Bootstrap starting.
I20251023 02:03:13.694819 21133 tablet_bootstrap.cc:492] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415: No bootstrap required, opened a new log
I20251023 02:03:13.695312 21133 ts_tablet_manager.cc:1403] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20251023 02:03:13.698118 21133 raft_consensus.cc:359] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:13.698850 21133 raft_consensus.cc:385] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.699143 21133 raft_consensus.cc:740] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.699849 21133 consensus_queue.cc:260] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:13.701433 21131 tablet_bootstrap.cc:654] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.703751 21133 ts_tablet_manager.cc:1434] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20251023 02:03:13.704412 21134 tablet_bootstrap.cc:492] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012: No bootstrap required, opened a new log
I20251023 02:03:13.707208 21134 ts_tablet_manager.cc:1403] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.011s
I20251023 02:03:13.709990 21134 raft_consensus.cc:359] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.710723 21134 raft_consensus.cc:385] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.711024 21134 raft_consensus.cc:740] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.711731 21134 consensus_queue.cc:260] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.714408 21133 tablet_bootstrap.cc:492] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415: Bootstrap starting.
I20251023 02:03:13.716594 21131 tablet_bootstrap.cc:492] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3: No bootstrap required, opened a new log
I20251023 02:03:13.717052 21131 ts_tablet_manager.cc:1403] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.003s
I20251023 02:03:13.719089 21134 ts_tablet_manager.cc:1434] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20251023 02:03:13.720048 21134 tablet_bootstrap.cc:492] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012: Bootstrap starting.
I20251023 02:03:13.719964 21131 raft_consensus.cc:359] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.720813 21133 tablet_bootstrap.cc:654] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.720913 21131 raft_consensus.cc:385] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.721314 21131 raft_consensus.cc:740] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.722242 21131 consensus_queue.cc:260] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.724754 21131 ts_tablet_manager.cc:1434] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20251023 02:03:13.725708 21131 tablet_bootstrap.cc:492] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3: Bootstrap starting.
I20251023 02:03:13.728502 21134 tablet_bootstrap.cc:654] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.732002 21131 tablet_bootstrap.cc:654] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.742731 21133 tablet_bootstrap.cc:492] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415: No bootstrap required, opened a new log
I20251023 02:03:13.743320 21133 ts_tablet_manager.cc:1403] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.015s
I20251023 02:03:13.746335 21133 raft_consensus.cc:359] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.747259 21133 raft_consensus.cc:385] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.747620 21133 raft_consensus.cc:740] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.748433 21133 consensus_queue.cc:260] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.752821 21133 ts_tablet_manager.cc:1434] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20251023 02:03:13.754120 21133 tablet_bootstrap.cc:492] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415: Bootstrap starting.
I20251023 02:03:13.758270 21138 raft_consensus.cc:493] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:13.758822 21138 raft_consensus.cc:515] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.761160 21134 tablet_bootstrap.cc:492] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012: No bootstrap required, opened a new log
I20251023 02:03:13.761706 21134 ts_tablet_manager.cc:1403] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.014s
I20251023 02:03:13.764624 21133 tablet_bootstrap.cc:654] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:13.764560 21134 raft_consensus.cc:359] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.765355 21134 raft_consensus.cc:385] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.765707 21134 raft_consensus.cc:740] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.766460 21134 consensus_queue.cc:260] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.770807 21138 leader_election.cc:290] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:13.774348 21139 raft_consensus.cc:493] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:13.774861 21139 raft_consensus.cc:515] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.780979 21139 leader_election.cc:290] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:13.796159 21135 raft_consensus.cc:493] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:13.796780 21135 raft_consensus.cc:515] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.799409 21135 leader_election.cc:290] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:13.758286 21131 tablet_bootstrap.cc:492] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3: No bootstrap required, opened a new log
I20251023 02:03:13.830361 21131 ts_tablet_manager.cc:1403] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3: Time spent bootstrapping tablet: real 0.105s user 0.012s sys 0.003s
I20251023 02:03:13.863193 21138 raft_consensus.cc:493] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:13.863735 21138 raft_consensus.cc:515] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.865504 21138 leader_election.cc:290] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:13.873039 21139 raft_consensus.cc:493] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:13.873505 21139 raft_consensus.cc:515] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.875530 21139 leader_election.cc:290] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:13.880908 21138 raft_consensus.cc:493] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:13.881490 21138 raft_consensus.cc:515] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.883826 21138 leader_election.cc:290] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:13.899935 21135 raft_consensus.cc:493] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:13.900465 21135 raft_consensus.cc:515] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.902420 21135 leader_election.cc:290] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:13.872148 21131 raft_consensus.cc:359] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:13.905623 21131 raft_consensus.cc:385] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:13.905994 21131 raft_consensus.cc:740] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Initialized, Role: FOLLOWER
I20251023 02:03:13.906860 21131 consensus_queue.cc:260] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:13.917966 21131 ts_tablet_manager.cc:1434] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3: Time spent starting tablet: real 0.087s user 0.006s sys 0.001s
I20251023 02:03:13.940598 21135 raft_consensus.cc:493] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:13.978339 21087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d54367d776794a079ca3bf28b8194efc" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012" is_pre_election: true
I20251023 02:03:13.946607 21139 raft_consensus.cc:493] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:13.955641 20938 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d54367d776794a079ca3bf28b8194efc" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3" is_pre_election: true
I20251023 02:03:13.979156 21087 raft_consensus.cc:2468] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 0.
I20251023 02:03:13.979094 21139 raft_consensus.cc:515] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.952219 20939 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3" is_pre_election: true
I20251023 02:03:13.980316 20939 raft_consensus.cc:2468] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 0.
I20251023 02:03:13.960924 20937 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3" is_pre_election: true
W20251023 02:03:13.980788 20978 leader_election.cc:343] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:13.981047 21139 leader_election.cc:290] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:13.981180 20937 raft_consensus.cc:2468] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 0.
I20251023 02:03:13.962185 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d54367d776794a079ca3bf28b8194efc" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3" is_pre_election: true
I20251023 02:03:13.976855 21088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012" is_pre_election: true
I20251023 02:03:13.982529 21088 raft_consensus.cc:2468] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 0.
I20251023 02:03:13.978168 21135 raft_consensus.cc:515] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.983494 20979 leader_election.cc:304] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415; no voters: 0130a1960e0740469b5291134a4e78a3
I20251023 02:03:13.983639 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6609dd61840640d796684211f3164998" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3" is_pre_election: true
I20251023 02:03:13.984593 21135 leader_election.cc:290] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:13.959367 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3" is_pre_election: true
I20251023 02:03:13.985327 21088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6609dd61840640d796684211f3164998" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012" is_pre_election: true
I20251023 02:03:13.985414 20935 raft_consensus.cc:2468] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 0.
I20251023 02:03:13.985956 21088 raft_consensus.cc:2468] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 0.
I20251023 02:03:13.986749 21139 raft_consensus.cc:2804] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:13.985451 20978 leader_election.cc:304] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:13.987134 21139 raft_consensus.cc:493] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:13.991575 21139 raft_consensus.cc:3060] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:13.993885 20979 leader_election.cc:304] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:14.001699 21168 raft_consensus.cc:2804] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251023 02:03:14.001801 20978 leader_election.cc:343] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:13.996095 21131 tablet_bootstrap.cc:492] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3: Bootstrap starting.
I20251023 02:03:14.002209 21168 raft_consensus.cc:493] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:14.009891 21168 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.013957 21169 raft_consensus.cc:2804] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:14.014369 21169 raft_consensus.cc:493] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:14.014679 21169 raft_consensus.cc:3060] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.019156 21138 raft_consensus.cc:493] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:14.019649 21138 raft_consensus.cc:515] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.021502 21138 leader_election.cc:290] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:14.023814 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.024191 21013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.024847 21013 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 0.
I20251023 02:03:14.024994 21139 raft_consensus.cc:515] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:13.991143 21052 leader_election.cc:304] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
W20251023 02:03:14.026809 20903 leader_election.cc:343] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:14.033566 20903 leader_election.cc:304] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:14.034103 21012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d54367d776794a079ca3bf28b8194efc" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.034323 21139 leader_election.cc:290] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.034310 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.034490 21135 raft_consensus.cc:2804] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:14.034934 21135 raft_consensus.cc:493] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:14.035116 21131 tablet_bootstrap.cc:654] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:14.035286 21135 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.035640 21011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.036446 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d54367d776794a079ca3bf28b8194efc" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3"
W20251023 02:03:14.037860 20978 leader_election.cc:343] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:14.028112 21134 ts_tablet_manager.cc:1434] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012: Time spent starting tablet: real 0.266s user 0.080s sys 0.146s
I20251023 02:03:14.020756 21168 raft_consensus.cc:515] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.039263 21014 raft_consensus.cc:2393] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1796d6cb245b41a29d0b808df3338012 in current term 1: Already voted for candidate 30052fa1e09b406e97bd508d13cac415 in this term.
I20251023 02:03:14.040115 21168 leader_election.cc:290] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.030581 21133 tablet_bootstrap.cc:492] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415: No bootstrap required, opened a new log
I20251023 02:03:14.040946 21133 ts_tablet_manager.cc:1403] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415: Time spent bootstrapping tablet: real 0.287s user 0.079s sys 0.170s
I20251023 02:03:14.042829 21088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d54367d776794a079ca3bf28b8194efc" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.043293 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:14.043890 20935 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.027424 21013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.043408 21088 raft_consensus.cc:3060] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.044037 21133 raft_consensus.cc:359] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.044857 21133 raft_consensus.cc:385] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:14.045161 21133 raft_consensus.cc:740] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Initialized, Role: FOLLOWER
I20251023 02:03:14.035118 21012 raft_consensus.cc:2393] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1796d6cb245b41a29d0b808df3338012 in current term 1: Already voted for candidate 30052fa1e09b406e97bd508d13cac415 in this term.
I20251023 02:03:14.045850 21133 consensus_queue.cc:260] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.048501 21133 ts_tablet_manager.cc:1434] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20251023 02:03:14.048838 21013 raft_consensus.cc:2468] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 0.
I20251023 02:03:14.049484 21133 tablet_bootstrap.cc:492] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415: Bootstrap starting.
I20251023 02:03:14.050616 20903 leader_election.cc:304] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:14.054334 20935 raft_consensus.cc:2468] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 1.
I20251023 02:03:14.058401 21171 raft_consensus.cc:2804] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:14.058831 21171 raft_consensus.cc:493] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:14.058996 21133 tablet_bootstrap.cc:654] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:14.058890 20978 leader_election.cc:304] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:14.059218 21171 raft_consensus.cc:3060] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.059862 21168 raft_consensus.cc:2804] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:14.060400 21168 raft_consensus.cc:697] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:14.061434 21168 consensus_queue.cc:237] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
W20251023 02:03:14.066404 21052 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:35337, user_credentials={real_user=slave}} blocked reactor thread for 75466.7us
W20251023 02:03:14.067158 21052 leader_election.cc:343] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:14.057871 21134 tablet_bootstrap.cc:492] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012: Bootstrap starting.
I20251023 02:03:14.044018 21087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.068281 21087 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.075210 21087 raft_consensus.cc:2468] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 1.
I20251023 02:03:14.030823 21169 raft_consensus.cc:515] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.065738 21088 raft_consensus.cc:2468] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 1.
I20251023 02:03:14.079476 20979 leader_election.cc:304] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415; no voters: 0130a1960e0740469b5291134a4e78a3
I20251023 02:03:14.080291 21139 raft_consensus.cc:2804] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:14.080519 21138 raft_consensus.cc:493] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:14.081036 21138 raft_consensus.cc:515] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.077078 21157 raft_consensus.cc:2764] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251023 02:03:14.067780 21052 leader_election.cc:304] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
I20251023 02:03:14.091681 21157 raft_consensus.cc:2804] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:14.089308 21135 raft_consensus.cc:515] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.094460 21157 raft_consensus.cc:493] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:14.096303 21138 leader_election.cc:290] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:14.097558 21157 raft_consensus.cc:3060] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.100715 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3" is_pre_election: true
I20251023 02:03:14.101429 21133 tablet_bootstrap.cc:492] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415: No bootstrap required, opened a new log
I20251023 02:03:14.101953 21133 ts_tablet_manager.cc:1403] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.011s
I20251023 02:03:14.101783 21013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.102227 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6609dd61840640d796684211f3164998" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:14.106292 21169 leader_election.cc:290] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.117229 21135 leader_election.cc:290] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 election: Requested vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
W20251023 02:03:14.126829 21052 leader_election.cc:343] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:14.128820 21052 leader_election.cc:304] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012; no voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:14.131692 21138 raft_consensus.cc:493] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:03:14.119557 20978 leader_election.cc:343] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:14.134117 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3" is_pre_election: true
I20251023 02:03:14.134946 20936 raft_consensus.cc:2393] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1796d6cb245b41a29d0b808df3338012 in current term 1: Already voted for candidate 0130a1960e0740469b5291134a4e78a3 in this term.
I20251023 02:03:14.133757 21138 raft_consensus.cc:515] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
W20251023 02:03:14.146466 21052 leader_election.cc:343] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:14.146828 21169 raft_consensus.cc:493] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:14.147317 21169 raft_consensus.cc:515] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:14.149150 21087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012" is_pre_election: true
I20251023 02:03:14.149516 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c36810008544d7fa5c781cae7b1d884" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3" is_pre_election: true
I20251023 02:03:14.150605 21134 tablet_bootstrap.cc:654] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:14.150812 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012" is_pre_election: true
I20251023 02:03:14.151417 21086 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 0.
I20251023 02:03:14.152540 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012" is_pre_election: true
W20251023 02:03:14.152786 21052 leader_election.cc:343] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:14.153203 21052 leader_election.cc:304] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012; no voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:14.153527 21087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.154107 21087 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.133708 21088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6609dd61840640d796684211f3164998" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.163484 21138 leader_election.cc:290] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:14.158167 21013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:14.164789 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3" is_pre_election: true
I20251023 02:03:14.158641 21139 raft_consensus.cc:697] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:14.164067 21088 raft_consensus.cc:3060] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.165508 21169 leader_election.cc:290] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.169390 21013 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.169759 21138 raft_consensus.cc:2749] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:03:14.171895 21131 tablet_bootstrap.cc:492] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3: No bootstrap required, opened a new log
W20251023 02:03:14.175824 21173 log.cc:927] Time spent T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20251023 02:03:14.176575 21131 ts_tablet_manager.cc:1403] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3: Time spent bootstrapping tablet: real 0.181s user 0.011s sys 0.029s
I20251023 02:03:14.175989 21139 consensus_queue.cc:237] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.175448 21085 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012" is_pre_election: true
I20251023 02:03:14.180030 21133 raft_consensus.cc:359] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.180948 21133 raft_consensus.cc:385] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:14.181227 21133 raft_consensus.cc:740] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Initialized, Role: FOLLOWER
W20251023 02:03:14.182418 20979 leader_election.cc:343] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:14.182046 21133 consensus_queue.cc:260] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.182871 21131 raft_consensus.cc:359] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.183557 21131 raft_consensus.cc:385] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:14.183821 21131 raft_consensus.cc:740] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Initialized, Role: FOLLOWER
W20251023 02:03:14.185036 21052 leader_election.cc:343] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337): Illegal state: must be running to vote when last-logged opid is not known
W20251023 02:03:14.187245 20904 leader_election.cc:343] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:14.187824 20904 leader_election.cc:304] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3; no voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:14.190914 21134 tablet_bootstrap.cc:492] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012: No bootstrap required, opened a new log
I20251023 02:03:14.191217 21134 ts_tablet_manager.cc:1403] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012: Time spent bootstrapping tablet: real 0.134s user 0.018s sys 0.010s
I20251023 02:03:14.190600 21135 raft_consensus.cc:2749] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:03:14.194010 21013 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 1.
I20251023 02:03:14.194633 21088 raft_consensus.cc:2468] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 1.
I20251023 02:03:14.195099 20903 leader_election.cc:304] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:14.195828 21135 raft_consensus.cc:2804] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:14.196352 20979 leader_election.cc:304] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415; no voters: 0130a1960e0740469b5291134a4e78a3
I20251023 02:03:14.198021 21087 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 1.
I20251023 02:03:14.198576 21169 raft_consensus.cc:2804] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:14.199936 21169 raft_consensus.cc:697] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:14.198130 21012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c36810008544d7fa5c781cae7b1d884" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.188095 21171 raft_consensus.cc:515] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.194034 21175 raft_consensus.cc:2749] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:03:14.190800 21131 consensus_queue.cc:260] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.203119 21157 raft_consensus.cc:515] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.207957 21135 raft_consensus.cc:697] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
W20251023 02:03:14.207749 21052 leader_election.cc:343] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:14.208572 21133 ts_tablet_manager.cc:1434] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415: Time spent starting tablet: real 0.106s user 0.009s sys 0.003s
I20251023 02:03:14.208635 21169 consensus_queue.cc:237] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.214809 21086 raft_consensus.cc:2393] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0130a1960e0740469b5291134a4e78a3 in current term 1: Already voted for candidate 1796d6cb245b41a29d0b808df3338012 in this term.
I20251023 02:03:14.215546 21171 leader_election.cc:290] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 election: Requested vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.216802 21012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:14.217379 21012 raft_consensus.cc:3060] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.210855 21052 leader_election.cc:304] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012; no voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:14.218515 21135 consensus_queue.cc:237] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.221609 21131 ts_tablet_manager.cc:1434] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3: Time spent starting tablet: real 0.043s user 0.006s sys 0.001s
I20251023 02:03:14.222671 21131 tablet_bootstrap.cc:492] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3: Bootstrap starting.
I20251023 02:03:14.223102 21133 tablet_bootstrap.cc:492] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415: Bootstrap starting.
I20251023 02:03:14.225380 21012 raft_consensus.cc:2468] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 1.
I20251023 02:03:14.226891 21157 leader_election.cc:290] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:14.228097 20903 leader_election.cc:304] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:14.227603 20834 catalog_manager.cc:5649] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:14.228951 21171 raft_consensus.cc:2804] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:14.229403 21171 raft_consensus.cc:697] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:14.230362 21171 consensus_queue.cc:237] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.231531 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.232295 21014 raft_consensus.cc:2393] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1796d6cb245b41a29d0b808df3338012 in current term 1: Already voted for candidate 0130a1960e0740469b5291134a4e78a3 in this term.
I20251023 02:03:14.234956 20936 raft_consensus.cc:2468] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 0.
I20251023 02:03:14.235334 21131 tablet_bootstrap.cc:654] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:14.236689 20978 leader_election.cc:304] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters: 1796d6cb245b41a29d0b808df3338012
I20251023 02:03:14.237623 21139 raft_consensus.cc:2804] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:14.237994 21139 raft_consensus.cc:493] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:14.238337 21139 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.246634 21134 raft_consensus.cc:359] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:14.248327 21134 raft_consensus.cc:385] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:14.248687 21134 raft_consensus.cc:740] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Initialized, Role: FOLLOWER
I20251023 02:03:14.252265 21052 leader_election.cc:304] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012; no voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:14.253958 21133 tablet_bootstrap.cc:654] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:14.252393 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:14.254539 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.249850 21134 consensus_queue.cc:260] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:14.247263 21157 raft_consensus.cc:2749] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:03:14.257792 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:14.258558 21014 raft_consensus.cc:2393] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1796d6cb245b41a29d0b808df3338012 in current term 1: Already voted for candidate 0130a1960e0740469b5291134a4e78a3 in this term.
I20251023 02:03:14.258886 21134 ts_tablet_manager.cc:1434] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012: Time spent starting tablet: real 0.067s user 0.008s sys 0.000s
I20251023 02:03:14.256565 21175 raft_consensus.cc:2749] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:03:14.259975 21134 tablet_bootstrap.cc:492] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012: Bootstrap starting.
I20251023 02:03:14.260910 21086 raft_consensus.cc:2393] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0130a1960e0740469b5291134a4e78a3 in current term 1: Already voted for candidate 1796d6cb245b41a29d0b808df3338012 in this term.
I20251023 02:03:14.270076 21052 leader_election.cc:304] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012; no voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:14.273350 21139 raft_consensus.cc:515] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:14.285300 21157 raft_consensus.cc:2749] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 02:03:14.288076 21131 tablet_bootstrap.cc:492] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3: No bootstrap required, opened a new log
I20251023 02:03:14.290486 21139 leader_election.cc:290] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.291337 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:14.292061 20936 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.293920 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.294544 21086 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.298928 21133 tablet_bootstrap.cc:492] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415: No bootstrap required, opened a new log
I20251023 02:03:14.302040 21133 ts_tablet_manager.cc:1403] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415: Time spent bootstrapping tablet: real 0.079s user 0.012s sys 0.008s
I20251023 02:03:14.302610 21134 tablet_bootstrap.cc:654] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:14.304894 21133 raft_consensus.cc:359] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.305548 21133 raft_consensus.cc:385] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:14.305845 21133 raft_consensus.cc:740] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Initialized, Role: FOLLOWER
I20251023 02:03:14.306537 21133 consensus_queue.cc:260] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.296415 21131 ts_tablet_manager.cc:1403] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.009s
I20251023 02:03:14.311159 21086 raft_consensus.cc:2468] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 1.
I20251023 02:03:14.312356 20979 leader_election.cc:304] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:14.313324 21139 raft_consensus.cc:2804] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:14.312361 20936 raft_consensus.cc:2468] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 1.
I20251023 02:03:14.313920 21139 raft_consensus.cc:697] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:14.315034 21139 consensus_queue.cc:237] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:14.315913 21133 ts_tablet_manager.cc:1434] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415: Time spent starting tablet: real 0.009s user 0.001s sys 0.005s
I20251023 02:03:14.317670 21131 raft_consensus.cc:359] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.318063 21133 tablet_bootstrap.cc:492] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415: Bootstrap starting.
I20251023 02:03:14.321166 21131 raft_consensus.cc:385] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:14.321934 21131 raft_consensus.cc:740] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Initialized, Role: FOLLOWER
I20251023 02:03:14.322716 21131 consensus_queue.cc:260] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.326756 21133 tablet_bootstrap.cc:654] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:14.328634 21134 tablet_bootstrap.cc:492] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012: No bootstrap required, opened a new log
I20251023 02:03:14.329211 21134 ts_tablet_manager.cc:1403] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.001s
I20251023 02:03:14.332525 21134 raft_consensus.cc:359] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.333436 21134 raft_consensus.cc:385] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:14.333820 21134 raft_consensus.cc:740] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Initialized, Role: FOLLOWER
I20251023 02:03:14.334264 21131 ts_tablet_manager.cc:1434] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20251023 02:03:14.334662 21134 consensus_queue.cc:260] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.336367 21134 ts_tablet_manager.cc:1434] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251023 02:03:14.338268 21131 tablet_bootstrap.cc:492] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3: Bootstrap starting.
I20251023 02:03:14.345695 21133 tablet_bootstrap.cc:492] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415: No bootstrap required, opened a new log
I20251023 02:03:14.346222 21133 ts_tablet_manager.cc:1403] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.005s
I20251023 02:03:14.347342 21131 tablet_bootstrap.cc:654] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:14.353214 21131 tablet_bootstrap.cc:492] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3: No bootstrap required, opened a new log
I20251023 02:03:14.348850 21133 raft_consensus.cc:359] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.353668 21133 raft_consensus.cc:385] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:14.353896 21131 ts_tablet_manager.cc:1403] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20251023 02:03:14.353950 21133 raft_consensus.cc:740] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Initialized, Role: FOLLOWER
I20251023 02:03:14.354818 21133 consensus_queue.cc:260] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.356350 21131 raft_consensus.cc:359] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.355855 20833 catalog_manager.cc:5649] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0130a1960e0740469b5291134a4e78a3 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "0130a1960e0740469b5291134a4e78a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:14.356848 21133 ts_tablet_manager.cc:1434] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251023 02:03:14.357002 21131 raft_consensus.cc:385] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:14.357287 21131 raft_consensus.cc:740] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Initialized, Role: FOLLOWER
I20251023 02:03:14.358405 21131 consensus_queue.cc:260] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.363025 21131 ts_tablet_manager.cc:1434] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3: Time spent starting tablet: real 0.009s user 0.001s sys 0.004s
I20251023 02:03:14.363821 21131 tablet_bootstrap.cc:492] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3: Bootstrap starting.
W20251023 02:03:14.365756 20884 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251023 02:03:14.369236 21131 tablet_bootstrap.cc:654] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:14.369863 20834 catalog_manager.cc:5649] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:14.372056 20834 catalog_manager.cc:5649] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:14.373898 20834 catalog_manager.cc:5649] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:14.381719 21131 tablet_bootstrap.cc:492] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3: No bootstrap required, opened a new log
I20251023 02:03:14.382184 21131 ts_tablet_manager.cc:1403] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.004s
I20251023 02:03:14.384559 21131 raft_consensus.cc:359] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.385196 21131 raft_consensus.cc:385] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:14.385466 21131 raft_consensus.cc:740] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Initialized, Role: FOLLOWER
I20251023 02:03:14.386153 21131 consensus_queue.cc:260] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.386121 20835 catalog_manager.cc:5649] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0130a1960e0740469b5291134a4e78a3 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "0130a1960e0740469b5291134a4e78a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:14.388299 21131 ts_tablet_manager.cc:1434] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251023 02:03:14.464756 21139 consensus_queue.cc:1048] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.483693 21139 consensus_queue.cc:1048] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.487242 21157 raft_consensus.cc:493] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:14.487746 21157 raft_consensus.cc:515] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.489555 21157 leader_election.cc:290] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:14.491357 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3" is_pre_election: true
I20251023 02:03:14.492798 20935 raft_consensus.cc:2468] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 0.
I20251023 02:03:14.491673 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.494194 21014 raft_consensus.cc:2468] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 0.
I20251023 02:03:14.495657 21052 leader_election.cc:304] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
I20251023 02:03:14.496943 21157 raft_consensus.cc:2804] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:14.497282 21157 raft_consensus.cc:493] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:14.497665 21157 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.506265 21157 raft_consensus.cc:515] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.508147 21157 leader_election.cc:290] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:14.509198 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:14.509191 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:14.510026 20936 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.510076 21014 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.510702 21171 raft_consensus.cc:493] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:14.511734 21171 raft_consensus.cc:515] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.513648 21171 leader_election.cc:290] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.514720 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012" is_pre_election: true
I20251023 02:03:14.515120 21012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.515336 21086 raft_consensus.cc:2468] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 0.
I20251023 02:03:14.515748 21012 raft_consensus.cc:2468] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 0.
I20251023 02:03:14.516423 20904 leader_election.cc:304] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
I20251023 02:03:14.517454 21171 raft_consensus.cc:2804] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:14.517818 21171 raft_consensus.cc:493] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:14.518165 21171 raft_consensus.cc:3060] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.518494 20936 raft_consensus.cc:2468] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 1.
I20251023 02:03:14.519722 21052 leader_election.cc:304] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
I20251023 02:03:14.519706 21014 raft_consensus.cc:2468] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 1.
I20251023 02:03:14.520573 21157 raft_consensus.cc:2804] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:14.521533 21157 raft_consensus.cc:697] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 1 LEADER]: Becoming Leader. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Running, Role: LEADER
I20251023 02:03:14.522382 21157 consensus_queue.cc:237] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.523519 21171 raft_consensus.cc:515] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.525729 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:14.525972 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.526403 21014 raft_consensus.cc:3060] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.526535 21086 raft_consensus.cc:3060] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.530481 21135 raft_consensus.cc:493] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:14.530982 21135 raft_consensus.cc:515] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.553494 21135 leader_election.cc:290] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.554420 21012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c36810008544d7fa5c781cae7b1d884" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.554526 21087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c36810008544d7fa5c781cae7b1d884" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012" is_pre_election: true
I20251023 02:03:14.555131 21012 raft_consensus.cc:2468] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 0.
I20251023 02:03:14.555202 21087 raft_consensus.cc:2468] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 0.
I20251023 02:03:14.555816 21014 raft_consensus.cc:2468] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 1.
I20251023 02:03:14.556533 20904 leader_election.cc:304] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
I20251023 02:03:14.559530 21135 raft_consensus.cc:2804] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:14.558290 20903 leader_election.cc:304] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:14.560160 21135 raft_consensus.cc:493] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:14.556972 21171 leader_election.cc:290] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 election: Requested vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.560910 21180 raft_consensus.cc:2804] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:14.561266 21180 raft_consensus.cc:697] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:14.562011 21180 consensus_queue.cc:237] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.560613 21135 raft_consensus.cc:3060] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.568933 20834 catalog_manager.cc:5649] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1796d6cb245b41a29d0b808df3338012 (127.20.81.3). New cstate: current_term: 1 leader_uuid: "1796d6cb245b41a29d0b808df3338012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:14.571617 21086 raft_consensus.cc:2468] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 1.
I20251023 02:03:14.573380 21135 raft_consensus.cc:515] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.573688 20835 catalog_manager.cc:5649] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0130a1960e0740469b5291134a4e78a3 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "0130a1960e0740469b5291134a4e78a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:14.575349 21135 leader_election.cc:290] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 election: Requested vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.575999 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c36810008544d7fa5c781cae7b1d884" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:14.576346 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c36810008544d7fa5c781cae7b1d884" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.576548 21014 raft_consensus.cc:3060] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.577005 21086 raft_consensus.cc:3060] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.581331 21014 raft_consensus.cc:2468] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 1.
I20251023 02:03:14.582396 20903 leader_election.cc:304] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:14.583225 21086 raft_consensus.cc:2468] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 1.
I20251023 02:03:14.584192 21135 raft_consensus.cc:2804] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:14.584658 21135 raft_consensus.cc:697] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:14.585460 21135 consensus_queue.cc:237] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.593300 20835 catalog_manager.cc:5649] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0130a1960e0740469b5291134a4e78a3 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "0130a1960e0740469b5291134a4e78a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:14.595995 21174 consensus_queue.cc:1048] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:03:14.605449 21169 consensus_queue.cc:1048] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.608119 20936 raft_consensus.cc:3060] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.619037 21174 consensus_queue.cc:1048] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.628371 21180 raft_consensus.cc:493] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:14.628849 21180 raft_consensus.cc:515] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.630713 21180 leader_election.cc:290] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.631805 21012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6609dd61840640d796684211f3164998" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30052fa1e09b406e97bd508d13cac415" is_pre_election: true
I20251023 02:03:14.636026 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6609dd61840640d796684211f3164998" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1796d6cb245b41a29d0b808df3338012" is_pre_election: true
I20251023 02:03:14.637275 20904 leader_election.cc:304] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3; no voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:14.638432 21171 consensus_queue.cc:1048] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.639534 21180 raft_consensus.cc:2749] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:03:14.653371 20935 raft_consensus.cc:3060] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:14.670306 21174 consensus_queue.cc:1048] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.670814 21171 consensus_queue.cc:1048] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.697963 21014 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
mode: GRACEFUL
new_leader_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:52082
I20251023 02:03:14.698698 21014 raft_consensus.cc:606] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Received request to transfer leadership to TS 0130a1960e0740469b5291134a4e78a3
I20251023 02:03:14.703626 21014 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
mode: GRACEFUL
new_leader_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:52082
I20251023 02:03:14.704066 21014 raft_consensus.cc:606] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Received request to transfer leadership to TS 0130a1960e0740469b5291134a4e78a3
I20251023 02:03:14.719396 21086 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7"
dest_uuid: "1796d6cb245b41a29d0b808df3338012"
mode: GRACEFUL
new_leader_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:36148
I20251023 02:03:14.720007 21086 raft_consensus.cc:606] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 1 LEADER]: Received request to transfer leadership to TS 0130a1960e0740469b5291134a4e78a3
I20251023 02:03:14.726719 21171 consensus_queue.cc:1048] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.740234 20935 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6c36810008544d7fa5c781cae7b1d884"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
mode: GRACEFUL
new_leader_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:51800
I20251023 02:03:14.740868 20935 raft_consensus.cc:606] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Received request to transfer leadership to TS 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:14.743743 21180 consensus_queue.cc:1048] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.752317 20935 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fb614549d596431bbedda15b128caf3e"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
mode: GRACEFUL
new_leader_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:51800
I20251023 02:03:14.752920 20935 raft_consensus.cc:606] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Received request to transfer leadership to TS 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:14.756107 21171 raft_consensus.cc:993] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3: : Instructing follower 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:14.756561 21171 raft_consensus.cc:1081] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Signalling peer 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:14.758654 21014 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fb614549d596431bbedda15b128caf3e"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:52068
I20251023 02:03:14.759127 21014 raft_consensus.cc:493] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:14.759524 21014 raft_consensus.cc:3060] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:14.759840 20935 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "be886e390b59440393859aef4d6e61f5"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
mode: GRACEFUL
new_leader_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:51800
I20251023 02:03:14.760382 20935 raft_consensus.cc:606] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Received request to transfer leadership to TS 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:14.765389 21014 raft_consensus.cc:515] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.769088 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:14.769639 20935 raft_consensus.cc:3055] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:14.769999 20935 raft_consensus.cc:740] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:14.771034 21014 leader_election.cc:290] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 2 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.772742 21014 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d54367d776794a079ca3bf28b8194efc"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
mode: GRACEFUL
new_leader_uuid: "1796d6cb245b41a29d0b808df3338012"
from {username='slave'} at 127.0.0.1:52082
I20251023 02:03:14.773466 21014 raft_consensus.cc:606] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Received request to transfer leadership to TS 1796d6cb245b41a29d0b808df3338012
I20251023 02:03:14.773398 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.771024 20935 consensus_queue.cc:260] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.774274 21086 raft_consensus.cc:3060] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:14.774866 20935 raft_consensus.cc:3060] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:14.780030 21086 raft_consensus.cc:2468] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 2.
I20251023 02:03:14.781083 20979 leader_election.cc:304] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:14.781638 20935 raft_consensus.cc:2468] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 2.
I20251023 02:03:14.781966 21174 raft_consensus.cc:2804] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:14.782444 21174 raft_consensus.cc:697] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:14.783236 21174 consensus_queue.cc:237] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.789994 20835 catalog_manager.cc:5649] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 1 to 2, leader changed from 0130a1960e0740469b5291134a4e78a3 (127.20.81.1) to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 2 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:14.857410 21174 consensus_queue.cc:1048] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.864204 21169 raft_consensus.cc:993] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415: : Instructing follower 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:14.864650 21169 raft_consensus.cc:1081] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Signalling peer 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:14.867017 20936 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:51796
I20251023 02:03:14.867584 20936 raft_consensus.cc:493] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:14.867945 20936 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:14.870322 21169 consensus_queue.cc:1048] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.875301 20936 raft_consensus.cc:515] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:14.878019 21087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.880354 20935 raft_consensus.cc:1240] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Rejecting Update request from peer 30052fa1e09b406e97bd508d13cac415 for earlier term 1. Current term is 2. Ops: []
I20251023 02:03:14.883075 20936 leader_election.cc:290] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 2 election: Requested vote from peers 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:14.883114 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:14.884080 21014 raft_consensus.cc:3055] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:14.884402 21014 raft_consensus.cc:740] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:14.885135 21014 consensus_queue.cc:260] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:14.886312 21014 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:14.891150 21014 raft_consensus.cc:2468] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 2.
I20251023 02:03:14.892153 20903 leader_election.cc:304] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters: 1796d6cb245b41a29d0b808df3338012
I20251023 02:03:14.892789 21171 raft_consensus.cc:2804] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:14.893236 21171 raft_consensus.cc:697] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:14.893929 21171 consensus_queue.cc:237] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:14.900405 20835 catalog_manager.cc:5649] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 reported cstate change: term changed from 1 to 2, leader changed from 30052fa1e09b406e97bd508d13cac415 (127.20.81.2) to 0130a1960e0740469b5291134a4e78a3 (127.20.81.1). New cstate: current_term: 2 leader_uuid: "0130a1960e0740469b5291134a4e78a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:14.920565 21169 raft_consensus.cc:993] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415: : Instructing follower 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:14.921082 21174 raft_consensus.cc:1081] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Signalling peer 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:14.922391 20935 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:51796
I20251023 02:03:14.922852 20935 raft_consensus.cc:493] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:14.923141 20935 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:14.927740 20935 raft_consensus.cc:515] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.929224 20935 leader_election.cc:290] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 2 election: Requested vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.930284 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:14.930303 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.930770 21014 raft_consensus.cc:3055] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:14.930953 21086 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:14.931092 21014 raft_consensus.cc:740] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:14.931799 21014 consensus_queue.cc:260] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.932648 21014 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:14.936935 21014 raft_consensus.cc:2468] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 2.
I20251023 02:03:14.937456 21086 raft_consensus.cc:2468] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 2.
I20251023 02:03:14.938047 20903 leader_election.cc:304] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:14.939219 21171 raft_consensus.cc:2804] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:14.939692 21171 raft_consensus.cc:697] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:14.940526 21171 consensus_queue.cc:237] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.947052 20835 catalog_manager.cc:5649] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 reported cstate change: term changed from 1 to 2, leader changed from 30052fa1e09b406e97bd508d13cac415 (127.20.81.2) to 0130a1960e0740469b5291134a4e78a3 (127.20.81.1). New cstate: current_term: 2 leader_uuid: "0130a1960e0740469b5291134a4e78a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:14.950922 21157 consensus_queue.cc:1048] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.961903 21157 consensus_queue.cc:1048] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:14.981036 21175 raft_consensus.cc:993] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012: : Instructing follower 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:14.981465 21219 raft_consensus.cc:1081] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 1 LEADER]: Signalling peer 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:14.982924 20935 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:51784
I20251023 02:03:14.983371 20935 raft_consensus.cc:493] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:14.983680 20935 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:14.988211 20935 raft_consensus.cc:515] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.989609 20935 leader_election.cc:290] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 2 election: Requested vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:14.990511 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:14.990897 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:14.991094 21014 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:14.991405 21086 raft_consensus.cc:3055] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:14.991731 21086 raft_consensus.cc:740] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Running, Role: LEADER
I20251023 02:03:14.992311 21086 consensus_queue.cc:260] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:14.993150 21086 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:14.997265 21086 raft_consensus.cc:2468] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 2.
I20251023 02:03:14.997608 21014 raft_consensus.cc:2468] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 2.
I20251023 02:03:14.998299 20904 leader_election.cc:304] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
I20251023 02:03:14.998993 21171 raft_consensus.cc:2804] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:14.999359 21171 raft_consensus.cc:697] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:15.000017 21171 consensus_queue.cc:237] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:15.006913 20833 catalog_manager.cc:5649] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 reported cstate change: term changed from 1 to 2, leader changed from 1796d6cb245b41a29d0b808df3338012 (127.20.81.3) to 0130a1960e0740469b5291134a4e78a3 (127.20.81.1). New cstate: current_term: 2 leader_uuid: "0130a1960e0740469b5291134a4e78a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:15.010093 21171 consensus_queue.cc:1048] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:15.018530 21179 raft_consensus.cc:993] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3: : Instructing follower 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:15.018959 21179 raft_consensus.cc:1081] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Signalling peer 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:15.020892 21012 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6c36810008544d7fa5c781cae7b1d884"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:52068
I20251023 02:03:15.021446 21012 raft_consensus.cc:493] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:15.023984 21012 raft_consensus.cc:3060] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:15.024883 21171 consensus_queue.cc:1048] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:15.030009 21012 raft_consensus.cc:515] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:15.031029 21014 raft_consensus.cc:1240] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Rejecting Update request from peer 0130a1960e0740469b5291134a4e78a3 for earlier term 1. Current term is 2. Ops: []
I20251023 02:03:15.032989 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c36810008544d7fa5c781cae7b1d884" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:15.033162 21012 leader_election.cc:290] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 2 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:15.037734 21180 consensus_queue.cc:1059] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:15.038551 21180 raft_consensus.cc:3055] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:15.038892 21180 raft_consensus.cc:740] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:15.039389 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c36810008544d7fa5c781cae7b1d884" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:15.039631 21180 consensus_queue.cc:260] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:15.041040 21180 raft_consensus.cc:3060] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:15.041637 21171 consensus_queue.cc:1048] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:15.059530 21180 consensus_queue.cc:1048] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:15.063938 20935 raft_consensus.cc:2468] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 2.
I20251023 02:03:15.065850 20978 leader_election.cc:304] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters: 1796d6cb245b41a29d0b808df3338012
I20251023 02:03:15.066728 21139 raft_consensus.cc:2804] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:15.067229 21139 raft_consensus.cc:697] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:15.067720 21169 raft_consensus.cc:993] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415: : Instructing follower 1796d6cb245b41a29d0b808df3338012 to start an election
I20251023 02:03:15.068145 21204 raft_consensus.cc:1081] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Signalling peer 1796d6cb245b41a29d0b808df3338012 to start an election
I20251023 02:03:15.068037 21139 consensus_queue.cc:237] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:15.069739 21086 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d54367d776794a079ca3bf28b8194efc"
dest_uuid: "1796d6cb245b41a29d0b808df3338012"
from {username='slave'} at 127.0.0.1:36138
I20251023 02:03:15.070272 21086 raft_consensus.cc:493] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:15.070688 21086 raft_consensus.cc:3060] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:15.078130 21086 raft_consensus.cc:515] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:15.079634 20835 catalog_manager.cc:5649] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 1 to 2, leader changed from 0130a1960e0740469b5291134a4e78a3 (127.20.81.1) to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 2 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:15.083482 21086 leader_election.cc:290] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 2 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:15.086760 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d54367d776794a079ca3bf28b8194efc" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:15.087334 21014 raft_consensus.cc:3055] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:15.087646 21014 raft_consensus.cc:740] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:15.088354 21014 consensus_queue.cc:260] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:15.089565 21014 raft_consensus.cc:3060] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:15.093899 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d54367d776794a079ca3bf28b8194efc" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:15.094516 20935 raft_consensus.cc:3060] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:15.100452 20935 raft_consensus.cc:2468] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 2.
I20251023 02:03:15.101378 21052 leader_election.cc:304] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
I20251023 02:03:15.102391 21219 raft_consensus.cc:2804] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:15.102898 21219 raft_consensus.cc:697] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 2 LEADER]: Becoming Leader. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Running, Role: LEADER
I20251023 02:03:15.103812 21014 raft_consensus.cc:2468] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 2.
I20251023 02:03:15.103749 21219 consensus_queue.cc:237] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:15.110913 20835 catalog_manager.cc:5649] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 reported cstate change: term changed from 1 to 2, leader changed from 30052fa1e09b406e97bd508d13cac415 (127.20.81.2) to 1796d6cb245b41a29d0b808df3338012 (127.20.81.3). New cstate: current_term: 2 leader_uuid: "1796d6cb245b41a29d0b808df3338012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:15.153757 21171 raft_consensus.cc:993] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3: : Instructing follower 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:15.154170 21180 raft_consensus.cc:1081] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Signalling peer 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:15.155584 21014 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "be886e390b59440393859aef4d6e61f5"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:52068
I20251023 02:03:15.156050 21014 raft_consensus.cc:493] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:15.156327 21014 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:15.162904 21014 raft_consensus.cc:515] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:15.165741 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:15.165761 21087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:15.166266 21014 leader_election.cc:290] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 2 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:15.166560 20935 raft_consensus.cc:3055] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:15.166630 21087 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:15.167115 20935 raft_consensus.cc:740] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:15.167937 20935 consensus_queue.cc:260] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:15.168797 20935 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:15.173125 20935 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 2.
I20251023 02:03:15.173653 21087 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 2.
I20251023 02:03:15.174139 20978 leader_election.cc:304] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:15.174897 21139 raft_consensus.cc:2804] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:15.175324 21139 raft_consensus.cc:697] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:15.176017 21139 consensus_queue.cc:237] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:15.182147 20835 catalog_manager.cc:5649] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 1 to 2, leader changed from 0130a1960e0740469b5291134a4e78a3 (127.20.81.1) to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 2 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:15.229540 21087 raft_consensus.cc:1275] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.230932 21139 consensus_queue.cc:1048] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:15.242117 20935 raft_consensus.cc:1275] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.243643 21139 consensus_queue.cc:1048] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:03:15.281018 21014 raft_consensus.cc:1275] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.282218 21180 consensus_queue.cc:1048] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:15.289743 21087 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:15.301693 21087 raft_consensus.cc:1275] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.303097 21180 consensus_queue.cc:1048] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:15.401515 21087 raft_consensus.cc:1275] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.402746 21180 consensus_queue.cc:1048] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:15.409832 21014 raft_consensus.cc:1275] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.411655 21171 consensus_queue.cc:1048] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:15.480475 21087 raft_consensus.cc:3060] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:15.485086 21087 raft_consensus.cc:1275] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.486407 21139 consensus_queue.cc:1048] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:15.491196 20936 raft_consensus.cc:1275] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Refusing update from remote peer 1796d6cb245b41a29d0b808df3338012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.493543 20936 raft_consensus.cc:1275] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.493714 21219 consensus_queue.cc:1048] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:03:15.495769 21169 consensus_queue.cc:1048] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:15.503716 21014 raft_consensus.cc:1275] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Refusing update from remote peer 1796d6cb245b41a29d0b808df3338012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.505158 21219 consensus_queue.cc:1048] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:15.595634 21014 raft_consensus.cc:1275] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.596850 21171 consensus_queue.cc:1048] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:15.597971 21087 raft_consensus.cc:1275] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.599081 21171 consensus_queue.cc:1048] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:15.738914 21087 raft_consensus.cc:1275] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.740002 21169 consensus_queue.cc:1048] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:15.747555 20935 raft_consensus.cc:1275] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:15.749071 21169 consensus_queue.cc:1048] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:17.804328 20804 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20251023 02:03:17.805088 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:17.824702 20935 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
mode: GRACEFUL
new_leader_uuid: "1796d6cb245b41a29d0b808df3338012"
from {username='slave'} at 127.0.0.1:51804
I20251023 02:03:17.825183 20935 raft_consensus.cc:606] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Received request to transfer leadership to TS 1796d6cb245b41a29d0b808df3338012
I20251023 02:03:17.834623 21014 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "be886e390b59440393859aef4d6e61f5"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
mode: GRACEFUL
new_leader_uuid: "1796d6cb245b41a29d0b808df3338012"
from {username='slave'} at 127.0.0.1:52088
I20251023 02:03:17.835078 21014 raft_consensus.cc:606] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Received request to transfer leadership to TS 1796d6cb245b41a29d0b808df3338012
I20251023 02:03:17.836010 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251023 02:03:17.836526 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:17.956005 21204 raft_consensus.cc:993] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415: : Instructing follower 1796d6cb245b41a29d0b808df3338012 to start an election
I20251023 02:03:17.956421 21234 raft_consensus.cc:1081] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Signalling peer 1796d6cb245b41a29d0b808df3338012 to start an election
I20251023 02:03:17.957758 21087 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "be886e390b59440393859aef4d6e61f5"
dest_uuid: "1796d6cb245b41a29d0b808df3338012"
from {username='slave'} at 127.0.0.1:36138
I20251023 02:03:17.958216 21087 raft_consensus.cc:493] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:17.958498 21087 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:17.962862 21087 raft_consensus.cc:515] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:17.964406 21087 leader_election.cc:290] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 3 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:17.965274 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:17.965467 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:17.966025 21014 raft_consensus.cc:3055] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:03:17.965943 20935 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:17.966362 21014 raft_consensus.cc:740] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:17.967051 21014 consensus_queue.cc:260] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:17.967900 21014 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:17.971477 20935 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 3.
I20251023 02:03:17.972127 21014 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 3.
I20251023 02:03:17.972651 21052 leader_election.cc:304] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
I20251023 02:03:17.973371 21229 raft_consensus.cc:2804] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:03:17.974416 21229 raft_consensus.cc:697] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Becoming Leader. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Running, Role: LEADER
I20251023 02:03:17.975044 21229 consensus_queue.cc:237] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:17.982743 20834 catalog_manager.cc:5649] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 reported cstate change: term changed from 2 to 3, leader changed from 30052fa1e09b406e97bd508d13cac415 (127.20.81.2) to 1796d6cb245b41a29d0b808df3338012 (127.20.81.3). New cstate: current_term: 3 leader_uuid: "1796d6cb245b41a29d0b808df3338012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:18.073993 21230 raft_consensus.cc:993] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3: : Instructing follower 1796d6cb245b41a29d0b808df3338012 to start an election
I20251023 02:03:18.074424 21241 raft_consensus.cc:1081] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Signalling peer 1796d6cb245b41a29d0b808df3338012 to start an election
I20251023 02:03:18.076153 21087 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7"
dest_uuid: "1796d6cb245b41a29d0b808df3338012"
from {username='slave'} at 127.0.0.1:36146
I20251023 02:03:18.076761 21087 raft_consensus.cc:493] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:18.077032 21087 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:18.081378 21087 raft_consensus.cc:515] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:18.082844 21087 leader_election.cc:290] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 3 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:18.084313 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:18.084584 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:18.084949 21014 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:18.085219 20935 raft_consensus.cc:3055] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:03:18.085623 20935 raft_consensus.cc:740] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:18.086426 20935 consensus_queue.cc:260] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:18.087733 20935 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:18.091545 21014 raft_consensus.cc:2468] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 3.
I20251023 02:03:18.092566 21052 leader_election.cc:304] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:18.093360 21229 raft_consensus.cc:2804] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:03:18.093902 21229 raft_consensus.cc:697] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Becoming Leader. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Running, Role: LEADER
I20251023 02:03:18.094732 21229 consensus_queue.cc:237] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:18.095686 20935 raft_consensus.cc:2468] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 3.
I20251023 02:03:18.101267 20834 catalog_manager.cc:5649] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 reported cstate change: term changed from 2 to 3, leader changed from 0130a1960e0740469b5291134a4e78a3 (127.20.81.1) to 1796d6cb245b41a29d0b808df3338012 (127.20.81.3). New cstate: current_term: 3 leader_uuid: "1796d6cb245b41a29d0b808df3338012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:18.360525 21014 raft_consensus.cc:1275] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Refusing update from remote peer 1796d6cb245b41a29d0b808df3338012: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:18.361799 21229 consensus_queue.cc:1048] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:03:18.371160 20935 raft_consensus.cc:1275] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Refusing update from remote peer 1796d6cb245b41a29d0b808df3338012: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:18.373224 21229 consensus_queue.cc:1048] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:03:18.538566 21014 raft_consensus.cc:1275] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Refusing update from remote peer 1796d6cb245b41a29d0b808df3338012: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:18.539677 21229 consensus_queue.cc:1048] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:03:18.548741 20935 raft_consensus.cc:1275] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Refusing update from remote peer 1796d6cb245b41a29d0b808df3338012: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:18.549883 21245 consensus_queue.cc:1048] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:03:18.837433 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:18.842344 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:18.842712 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:19.843559 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:19.849063 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:19.849455 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:20.850517 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:20.855268 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:20.855623 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:21.856804 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:21.862893 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:21.863287 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:22.864285 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:22.870012 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:22.870568 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:23.871412 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:23.876080 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:23.876425 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:24.877378 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:24.883131 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:24.883488 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:25.884260 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:25.888913 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:25.889249 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:26.890568 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:26.897485 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:26.897893 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:27.924721 20804 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20251023 02:03:27.928750 20936 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
mode: GRACEFUL
new_leader_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:51800
I20251023 02:03:27.929381 20936 raft_consensus.cc:606] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Received request to transfer leadership to TS 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:27.933490 20936 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
mode: GRACEFUL
new_leader_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:51800
I20251023 02:03:27.933967 20936 raft_consensus.cc:606] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Received request to transfer leadership to TS 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:27.937927 20936 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
mode: GRACEFUL
new_leader_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:51800
I20251023 02:03:27.938365 20936 raft_consensus.cc:606] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Received request to transfer leadership to TS 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:27.942471 21086 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7"
dest_uuid: "1796d6cb245b41a29d0b808df3338012"
mode: GRACEFUL
new_leader_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:36148
I20251023 02:03:27.942922 21086 raft_consensus.cc:606] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Received request to transfer leadership to TS 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:27.950909 21086 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "be886e390b59440393859aef4d6e61f5"
dest_uuid: "1796d6cb245b41a29d0b808df3338012"
mode: GRACEFUL
new_leader_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:36148
I20251023 02:03:27.951366 21086 raft_consensus.cc:606] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Received request to transfer leadership to TS 30052fa1e09b406e97bd508d13cac415
I20251023 02:03:28.103439 21261 raft_consensus.cc:993] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012: : Instructing follower 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:28.103953 21262 raft_consensus.cc:993] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3: : Instructing follower 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:28.104252 21258 raft_consensus.cc:1081] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Signalling peer 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:28.104595 21256 raft_consensus.cc:1081] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Signalling peer 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:28.105909 21012 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:52070
I20251023 02:03:28.106388 21014 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:52068
I20251023 02:03:28.106436 21012 raft_consensus.cc:493] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:28.106945 21012 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:28.106925 21014 raft_consensus.cc:493] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:28.107424 21014 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:28.111847 21014 raft_consensus.cc:515] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.112114 21012 raft_consensus.cc:515] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.113603 21014 leader_election.cc:290] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 3 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:28.113734 21012 leader_election.cc:290] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 4 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:28.114694 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:28.115160 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:28.115511 20936 raft_consensus.cc:3055] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:03:28.115653 20935 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:28.115742 21087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49e9de91bc8041e98d22df0219352ba7" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:28.115885 20936 raft_consensus.cc:740] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:28.114939 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:28.116387 21087 raft_consensus.cc:3055] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:03:28.116631 21086 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:28.116696 21087 raft_consensus.cc:740] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Running, Role: LEADER
I20251023 02:03:28.116843 20936 consensus_queue.cc:260] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.117489 21087 consensus_queue.cc:260] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.118342 20936 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:28.118775 21087 raft_consensus.cc:3060] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:28.122536 20935 raft_consensus.cc:2468] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 4.
I20251023 02:03:28.123740 21086 raft_consensus.cc:2468] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 3.
I20251023 02:03:28.124079 20978 leader_election.cc:304] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:28.125102 20979 leader_election.cc:304] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:28.125142 20936 raft_consensus.cc:2468] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 3.
I20251023 02:03:28.125253 21259 raft_consensus.cc:2804] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:03:28.125454 21087 raft_consensus.cc:2468] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 4.
I20251023 02:03:28.128516 21263 raft_consensus.cc:2804] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:03:28.129762 21263 raft_consensus.cc:697] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 3 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:28.130573 21259 raft_consensus.cc:697] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 4 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:28.131258 21263 consensus_queue.cc:237] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.131341 21259 consensus_queue.cc:237] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.132720 21261 raft_consensus.cc:993] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012: : Instructing follower 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:28.133142 21258 raft_consensus.cc:1081] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Signalling peer 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:28.134895 21012 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "be886e390b59440393859aef4d6e61f5"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:52070
I20251023 02:03:28.135721 21012 raft_consensus.cc:493] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:28.137378 21012 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:28.141175 20836 catalog_manager.cc:5649] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 3 to 4, leader changed from 1796d6cb245b41a29d0b808df3338012 (127.20.81.3) to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:28.142298 21012 raft_consensus.cc:515] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.142350 20836 catalog_manager.cc:5649] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 2 to 3, leader changed from 0130a1960e0740469b5291134a4e78a3 (127.20.81.1) to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 3 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:28.144266 21012 leader_election.cc:290] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 4 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:28.144569 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:28.144963 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:28.145257 20936 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:28.145557 21086 raft_consensus.cc:3055] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:03:28.145947 21086 raft_consensus.cc:740] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Running, Role: LEADER
I20251023 02:03:28.146777 21086 consensus_queue.cc:260] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.148123 21086 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:28.150487 20936 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 4.
I20251023 02:03:28.151644 20978 leader_election.cc:304] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:28.152494 21263 raft_consensus.cc:2804] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:03:28.152972 21263 raft_consensus.cc:697] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 4 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:28.154039 21086 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 4.
I20251023 02:03:28.154006 21263 consensus_queue.cc:237] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.163337 20836 catalog_manager.cc:5649] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 3 to 4, leader changed from 1796d6cb245b41a29d0b808df3338012 (127.20.81.3) to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:28.236166 21262 raft_consensus.cc:993] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3: : Instructing follower 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:28.236586 21256 raft_consensus.cc:1081] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Signalling peer 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:28.237879 21012 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:52068
I20251023 02:03:28.238297 21012 raft_consensus.cc:493] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:28.238557 21012 raft_consensus.cc:3060] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:28.242941 21012 raft_consensus.cc:515] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.244539 21012 leader_election.cc:290] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 2 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:28.245172 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:28.245482 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:28.245654 20936 raft_consensus.cc:3055] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:28.246008 20936 raft_consensus.cc:740] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:28.246022 21086 raft_consensus.cc:3060] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:28.246714 20936 consensus_queue.cc:260] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.247857 20936 raft_consensus.cc:3060] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:28.250736 21086 raft_consensus.cc:2468] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 2.
I20251023 02:03:28.251639 20979 leader_election.cc:304] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:28.252306 21263 raft_consensus.cc:2804] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:28.252405 20936 raft_consensus.cc:2468] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 2.
I20251023 02:03:28.252810 21263 raft_consensus.cc:697] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:28.253948 21262 raft_consensus.cc:993] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3: : Instructing follower 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:28.254323 21256 raft_consensus.cc:1081] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Signalling peer 30052fa1e09b406e97bd508d13cac415 to start an election
I20251023 02:03:28.254098 21263 consensus_queue.cc:237] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:28.255558 21012 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
from {username='slave'} at 127.0.0.1:52068
I20251023 02:03:28.256007 21012 raft_consensus.cc:493] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:28.256301 21012 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:28.261229 20836 catalog_manager.cc:5649] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 1 to 2, leader changed from 0130a1960e0740469b5291134a4e78a3 (127.20.81.1) to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 2 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:28.261667 21012 raft_consensus.cc:515] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:28.263305 21012 leader_election.cc:290] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 3 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:28.263516 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:28.263832 21086 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e" candidate_uuid: "30052fa1e09b406e97bd508d13cac415" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:28.264030 20936 raft_consensus.cc:3055] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:03:28.264287 21086 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:28.264303 20936 raft_consensus.cc:740] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:28.265086 20936 consensus_queue.cc:260] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:28.266011 20936 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:28.270273 21086 raft_consensus.cc:2468] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 3.
I20251023 02:03:28.271296 20979 leader_election.cc:304] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1796d6cb245b41a29d0b808df3338012, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:28.272228 21263 raft_consensus.cc:2804] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:03:28.272701 21263 raft_consensus.cc:697] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 3 LEADER]: Becoming Leader. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:28.272974 20936 raft_consensus.cc:2468] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30052fa1e09b406e97bd508d13cac415 in term 3.
I20251023 02:03:28.273677 21263 consensus_queue.cc:237] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:28.279970 20836 catalog_manager.cc:5649] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 reported cstate change: term changed from 2 to 3, leader changed from 0130a1960e0740469b5291134a4e78a3 (127.20.81.1) to 30052fa1e09b406e97bd508d13cac415 (127.20.81.2). New cstate: current_term: 3 leader_uuid: "30052fa1e09b406e97bd508d13cac415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:28.512742 20936 raft_consensus.cc:1275] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:28.513808 21263 consensus_queue.cc:1048] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:03:28.523553 21086 raft_consensus.cc:1275] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:28.525333 21263 consensus_queue.cc:1048] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:03:28.534097 21086 raft_consensus.cc:1275] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:03:28.535164 21259 consensus_queue.cc:1048] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:03:28.544255 20936 raft_consensus.cc:1275] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:03:28.545447 21259 consensus_queue.cc:1048] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:03:28.604839 20936 raft_consensus.cc:1275] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:03:28.605890 21259 consensus_queue.cc:1048] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:03:28.613638 21086 raft_consensus.cc:1275] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:03:28.615106 21265 consensus_queue.cc:1048] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:03:28.657836 21086 raft_consensus.cc:1275] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:28.658994 21265 consensus_queue.cc:1048] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:03:28.659597 21086 raft_consensus.cc:1275] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:28.660746 21265 consensus_queue.cc:1048] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:28.668668 20936 raft_consensus.cc:1275] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:28.670552 21263 consensus_queue.cc:1048] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:03:28.675034 20935 raft_consensus.cc:1275] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Refusing update from remote peer 30052fa1e09b406e97bd508d13cac415: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:28.677670 21265 consensus_queue.cc:1048] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:30.986958 20804 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20251023 02:03:30.987789 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:30.994678 21012 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
mode: GRACEFUL
new_leader_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:52088
I20251023 02:03:30.995291 21012 raft_consensus.cc:606] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Received request to transfer leadership to TS 0130a1960e0740469b5291134a4e78a3
I20251023 02:03:30.997680 21012 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
mode: GRACEFUL
new_leader_uuid: "1796d6cb245b41a29d0b808df3338012"
from {username='slave'} at 127.0.0.1:52088
I20251023 02:03:30.998255 21012 raft_consensus.cc:606] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 3 LEADER]: Received request to transfer leadership to TS 1796d6cb245b41a29d0b808df3338012
I20251023 02:03:31.000192 21012 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
mode: GRACEFUL
new_leader_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:52088
I20251023 02:03:31.000572 21012 raft_consensus.cc:606] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 3 LEADER]: Received request to transfer leadership to TS 0130a1960e0740469b5291134a4e78a3
I20251023 02:03:31.002454 21012 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "be886e390b59440393859aef4d6e61f5"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
mode: GRACEFUL
new_leader_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:52088
I20251023 02:03:31.002938 21012 raft_consensus.cc:606] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 4 LEADER]: Received request to transfer leadership to TS 0130a1960e0740469b5291134a4e78a3
I20251023 02:03:31.005448 21012 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fb614549d596431bbedda15b128caf3e"
dest_uuid: "30052fa1e09b406e97bd508d13cac415"
mode: GRACEFUL
new_leader_uuid: "1796d6cb245b41a29d0b808df3338012"
from {username='slave'} at 127.0.0.1:52088
I20251023 02:03:31.005949 21012 raft_consensus.cc:606] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Received request to transfer leadership to TS 1796d6cb245b41a29d0b808df3338012
I20251023 02:03:31.006791 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251023 02:03:31.007230 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:31.122387 21265 raft_consensus.cc:993] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415: : Instructing follower 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:31.122929 21288 raft_consensus.cc:1081] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 3 LEADER]: Signalling peer 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:31.124333 20936 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:51796
I20251023 02:03:31.124769 20936 raft_consensus.cc:493] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:31.125113 20936 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:31.148114 20936 raft_consensus.cc:515] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:31.150092 20936 leader_election.cc:290] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 4 election: Requested vote from peers 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:31.151341 21087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:31.151328 21012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69c64628ec8443168d18dbd1f66cc92e" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:31.152057 21087 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:31.152137 21012 raft_consensus.cc:3055] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:03:31.152626 21012 raft_consensus.cc:740] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:31.153417 21012 consensus_queue.cc:260] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:31.154587 21012 raft_consensus.cc:3060] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:31.181377 21265 raft_consensus.cc:993] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415: : Instructing follower 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:31.181841 21265 raft_consensus.cc:1081] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Signalling peer 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:31.183297 20936 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:51796
I20251023 02:03:31.183805 20936 raft_consensus.cc:493] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:31.184152 20936 raft_consensus.cc:3060] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:31.190771 21265 raft_consensus.cc:993] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415: : Instructing follower 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:31.191145 21288 raft_consensus.cc:1081] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 4 LEADER]: Signalling peer 0130a1960e0740469b5291134a4e78a3 to start an election
I20251023 02:03:31.192430 20935 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "be886e390b59440393859aef4d6e61f5"
dest_uuid: "0130a1960e0740469b5291134a4e78a3"
from {username='slave'} at 127.0.0.1:51796
I20251023 02:03:31.192812 20935 raft_consensus.cc:493] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:31.193050 20935 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:03:31.248018 21012 raft_consensus.cc:2468] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 4.
I20251023 02:03:31.248059 21087 raft_consensus.cc:2468] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 4.
I20251023 02:03:31.249511 20936 raft_consensus.cc:515] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.250069 20904 leader_election.cc:304] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
I20251023 02:03:31.251567 20936 leader_election.cc:290] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 3 election: Requested vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:31.253518 21012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:31.254037 21290 raft_consensus.cc:2804] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:03:31.254141 21012 raft_consensus.cc:3055] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:03:31.254848 21012 raft_consensus.cc:740] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:31.254803 21087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648a6aac62494cdda1eb0ff98736376b" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:31.255365 21290 raft_consensus.cc:697] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:31.255810 21012 consensus_queue.cc:260] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.255551 21087 raft_consensus.cc:3060] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:31.257213 21012 raft_consensus.cc:3060] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:31.257093 21290 consensus_queue.cc:237] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } }
I20251023 02:03:31.266876 20836 catalog_manager.cc:5649] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 reported cstate change: term changed from 3 to 4, leader changed from 30052fa1e09b406e97bd508d13cac415 (127.20.81.2) to 0130a1960e0740469b5291134a4e78a3 (127.20.81.1). New cstate: current_term: 4 leader_uuid: "0130a1960e0740469b5291134a4e78a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:31.352496 21012 raft_consensus.cc:2468] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 3.
I20251023 02:03:31.352495 21087 raft_consensus.cc:2468] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 3.
I20251023 02:03:31.352557 20935 raft_consensus.cc:515] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.354256 20903 leader_election.cc:304] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:31.354657 20935 leader_election.cc:290] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 5 election: Requested vote from peers 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407), 1796d6cb245b41a29d0b808df3338012 (127.20.81.3:33625)
I20251023 02:03:31.355460 21290 raft_consensus.cc:2804] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:03:31.356000 21290 raft_consensus.cc:697] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:31.356055 21087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "1796d6cb245b41a29d0b808df3338012"
I20251023 02:03:31.356448 21012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be886e390b59440393859aef4d6e61f5" candidate_uuid: "0130a1960e0740469b5291134a4e78a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:31.356647 21087 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:03:31.356979 21012 raft_consensus.cc:3055] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:03:31.357379 21012 raft_consensus.cc:740] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:31.357030 21290 consensus_queue.cc:237] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.358211 21012 consensus_queue.cc:260] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.359771 21012 raft_consensus.cc:3060] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:03:31.365398 20836 catalog_manager.cc:5649] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 reported cstate change: term changed from 2 to 3, leader changed from 30052fa1e09b406e97bd508d13cac415 (127.20.81.2) to 0130a1960e0740469b5291134a4e78a3 (127.20.81.1). New cstate: current_term: 3 leader_uuid: "0130a1960e0740469b5291134a4e78a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:31.395130 21265 raft_consensus.cc:993] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415: : Instructing follower 1796d6cb245b41a29d0b808df3338012 to start an election
I20251023 02:03:31.395545 21288 raft_consensus.cc:1081] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 3 LEADER]: Signalling peer 1796d6cb245b41a29d0b808df3338012 to start an election
I20251023 02:03:31.397123 21288 raft_consensus.cc:993] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415: : Instructing follower 1796d6cb245b41a29d0b808df3338012 to start an election
I20251023 02:03:31.397238 21086 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3"
dest_uuid: "1796d6cb245b41a29d0b808df3338012"
from {username='slave'} at 127.0.0.1:36138
I20251023 02:03:31.397645 21298 raft_consensus.cc:1081] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Signalling peer 1796d6cb245b41a29d0b808df3338012 to start an election
I20251023 02:03:31.397815 21086 raft_consensus.cc:493] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:31.398178 21086 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:31.399510 21088 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fb614549d596431bbedda15b128caf3e"
dest_uuid: "1796d6cb245b41a29d0b808df3338012"
from {username='slave'} at 127.0.0.1:36138
I20251023 02:03:31.399974 21088 raft_consensus.cc:493] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:31.400245 21088 raft_consensus.cc:3060] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:31.452924 21087 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 5.
I20251023 02:03:31.452874 21012 raft_consensus.cc:2468] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0130a1960e0740469b5291134a4e78a3 in term 5.
I20251023 02:03:31.454931 20903 leader_election.cc:304] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 30052fa1e09b406e97bd508d13cac415; no voters:
I20251023 02:03:31.455847 21290 raft_consensus.cc:2804] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:03:31.456303 21290 raft_consensus.cc:697] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0130a1960e0740469b5291134a4e78a3, State: Running, Role: LEADER
I20251023 02:03:31.457253 21290 consensus_queue.cc:237] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.465059 20836 catalog_manager.cc:5649] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 reported cstate change: term changed from 4 to 5, leader changed from 30052fa1e09b406e97bd508d13cac415 (127.20.81.2) to 0130a1960e0740469b5291134a4e78a3 (127.20.81.1). New cstate: current_term: 5 leader_uuid: "0130a1960e0740469b5291134a4e78a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:31.555575 21086 raft_consensus.cc:515] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.555696 21088 raft_consensus.cc:515] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.558328 21088 leader_election.cc:290] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 3 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:31.558526 21086 leader_election.cc:290] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 4 election: Requested vote from peers 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337), 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407)
I20251023 02:03:31.560104 20935 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:31.560334 21014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:31.560405 21012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb614549d596431bbedda15b128caf3e" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30052fa1e09b406e97bd508d13cac415"
I20251023 02:03:31.560685 20935 raft_consensus.cc:3060] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:31.560900 21014 raft_consensus.cc:3055] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:03:31.560969 21012 raft_consensus.cc:3055] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:03:31.561010 20936 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "676f88b0f9984cba93d7407ac7d88bd3" candidate_uuid: "1796d6cb245b41a29d0b808df3338012" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0130a1960e0740469b5291134a4e78a3"
I20251023 02:03:31.561390 21014 raft_consensus.cc:740] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:31.561497 21012 raft_consensus.cc:740] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30052fa1e09b406e97bd508d13cac415, State: Running, Role: LEADER
I20251023 02:03:31.561688 20936 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:31.562268 21014 consensus_queue.cc:260] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.562405 21012 consensus_queue.cc:260] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.563503 21014 raft_consensus.cc:3060] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:03:31.563630 21012 raft_consensus.cc:3060] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:03:31.649909 21013 raft_consensus.cc:1275] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:03:31.651283 21290 consensus_queue.cc:1048] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:03:31.657449 20935 raft_consensus.cc:2468] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 3.
I20251023 02:03:31.657711 20936 raft_consensus.cc:2468] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 4.
I20251023 02:03:31.658747 21052 leader_election.cc:304] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
I20251023 02:03:31.659318 21014 raft_consensus.cc:2468] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 4.
I20251023 02:03:31.659667 21052 leader_election.cc:304] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0130a1960e0740469b5291134a4e78a3, 1796d6cb245b41a29d0b808df3338012; no voters:
I20251023 02:03:31.660054 21012 raft_consensus.cc:2468] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1796d6cb245b41a29d0b808df3338012 in term 3.
I20251023 02:03:31.661108 21258 raft_consensus.cc:2804] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:03:31.662889 21303 raft_consensus.cc:2804] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:03:31.663311 21258 raft_consensus.cc:697] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Becoming Leader. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Running, Role: LEADER
I20251023 02:03:31.664505 21258 consensus_queue.cc:237] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.675323 21303 raft_consensus.cc:697] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 4 LEADER]: Becoming Leader. State: Replica: 1796d6cb245b41a29d0b808df3338012, State: Running, Role: LEADER
I20251023 02:03:31.675705 21086 raft_consensus.cc:1275] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:03:31.676219 21303 consensus_queue.cc:237] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } }
I20251023 02:03:31.677868 21290 consensus_queue.cc:1048] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:03:31.692884 20836 catalog_manager.cc:5649] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 reported cstate change: term changed from 2 to 3, leader changed from 30052fa1e09b406e97bd508d13cac415 (127.20.81.2) to 1796d6cb245b41a29d0b808df3338012 (127.20.81.3). New cstate: current_term: 3 leader_uuid: "1796d6cb245b41a29d0b808df3338012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:31.694382 20836 catalog_manager.cc:5649] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 reported cstate change: term changed from 3 to 4, leader changed from 30052fa1e09b406e97bd508d13cac415 (127.20.81.2) to 1796d6cb245b41a29d0b808df3338012 (127.20.81.3). New cstate: current_term: 4 leader_uuid: "1796d6cb245b41a29d0b808df3338012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:31.829272 21086 raft_consensus.cc:1275] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:31.830478 21290 consensus_queue.cc:1048] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:03:31.836571 21012 raft_consensus.cc:1275] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 5 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251023 02:03:31.841261 21014 raft_consensus.cc:1275] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:31.843389 21086 raft_consensus.cc:1275] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 5 FOLLOWER]: Refusing update from remote peer 0130a1960e0740469b5291134a4e78a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251023 02:03:31.843626 21290 consensus_queue.cc:1048] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:03:31.843713 21291 consensus_queue.cc:1048] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:03:31.845139 21291 consensus_queue.cc:1048] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1796d6cb245b41a29d0b808df3338012" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:03:32.008591 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:32.013999 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:32.014434 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:32.102515 20935 raft_consensus.cc:1275] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Refusing update from remote peer 1796d6cb245b41a29d0b808df3338012: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:32.103679 21258 consensus_queue.cc:1048] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:03:32.111600 21014 raft_consensus.cc:1275] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Refusing update from remote peer 1796d6cb245b41a29d0b808df3338012: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:03:32.113099 21304 consensus_queue.cc:1048] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:03:32.208777 21014 raft_consensus.cc:1275] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Refusing update from remote peer 1796d6cb245b41a29d0b808df3338012: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:03:32.210880 21304 consensus_queue.cc:1048] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30052fa1e09b406e97bd508d13cac415" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:03:32.222066 20935 raft_consensus.cc:1275] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Refusing update from remote peer 1796d6cb245b41a29d0b808df3338012: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:03:32.223371 21304 consensus_queue.cc:1048] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0130a1960e0740469b5291134a4e78a3" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:03:33.015390 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:33.020416 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:33.020805 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:34.021747 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:34.026270 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:34.026635 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:35.027657 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:35.032275 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:35.032639 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:36.033919 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:36.038892 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:36.039306 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:37.040217 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:37.046916 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:37.047362 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:38.048241 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:38.053542 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:38.053944 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:39.054872 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:39.059588 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:39.059962 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:40.061019 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:40.066933 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:40.067384 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:41.593974 20804 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20251023 02:03:41.594563 20804 tablet_server.cc:178] TabletServer@127.20.81.1:0 shutting down...
I20251023 02:03:41.624575 20804 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251023 02:03:41.625325 20804 tablet_replica.cc:333] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3: stopping tablet replica
I20251023 02:03:41.626039 20804 raft_consensus.cc:2243] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.626648 20804 raft_consensus.cc:2272] T 6609dd61840640d796684211f3164998 P 0130a1960e0740469b5291134a4e78a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.628926 20804 tablet_replica.cc:333] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3: stopping tablet replica
I20251023 02:03:41.629459 20804 raft_consensus.cc:2243] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.629976 20804 raft_consensus.cc:2272] T fb614549d596431bbedda15b128caf3e P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.631611 20804 tablet_replica.cc:333] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3: stopping tablet replica
I20251023 02:03:41.632083 20804 raft_consensus.cc:2243] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 4 LEADER]: Raft consensus shutting down.
I20251023 02:03:41.632856 20804 raft_consensus.cc:2272] T 69c64628ec8443168d18dbd1f66cc92e P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.634407 20804 tablet_replica.cc:333] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3: stopping tablet replica
I20251023 02:03:41.634862 20804 raft_consensus.cc:2243] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 3 LEADER]: Raft consensus shutting down.
I20251023 02:03:41.635572 20804 raft_consensus.cc:2272] T 648a6aac62494cdda1eb0ff98736376b P 0130a1960e0740469b5291134a4e78a3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.637053 20804 tablet_replica.cc:333] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3: stopping tablet replica
I20251023 02:03:41.637516 20804 raft_consensus.cc:2243] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.638048 20804 raft_consensus.cc:2272] T 49e9de91bc8041e98d22df0219352ba7 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.639590 20804 tablet_replica.cc:333] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3: stopping tablet replica
I20251023 02:03:41.640039 20804 raft_consensus.cc:2243] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.640432 20804 raft_consensus.cc:2272] T 6c36810008544d7fa5c781cae7b1d884 P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.641904 20804 tablet_replica.cc:333] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3: stopping tablet replica
I20251023 02:03:41.642350 20804 raft_consensus.cc:2243] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.642730 20804 raft_consensus.cc:2272] T 676f88b0f9984cba93d7407ac7d88bd3 P 0130a1960e0740469b5291134a4e78a3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.644241 20804 tablet_replica.cc:333] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3: stopping tablet replica
I20251023 02:03:41.644686 20804 raft_consensus.cc:2243] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.645097 20804 raft_consensus.cc:2272] T d54367d776794a079ca3bf28b8194efc P 0130a1960e0740469b5291134a4e78a3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.646576 20804 tablet_replica.cc:333] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3: stopping tablet replica
I20251023 02:03:41.646970 20804 raft_consensus.cc:2243] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 5 LEADER]: Raft consensus shutting down.
I20251023 02:03:41.647676 20804 raft_consensus.cc:2272] T be886e390b59440393859aef4d6e61f5 P 0130a1960e0740469b5291134a4e78a3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.728928 20804 tablet_server.cc:195] TabletServer@127.20.81.1:0 shutdown complete.
I20251023 02:03:41.748308 20804 tablet_server.cc:178] TabletServer@127.20.81.2:0 shutting down...
I20251023 02:03:41.772763 20804 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251023 02:03:41.773535 20804 tablet_replica.cc:333] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415: stopping tablet replica
I20251023 02:03:41.774550 20804 raft_consensus.cc:2243] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.775241 20804 raft_consensus.cc:2272] T 648a6aac62494cdda1eb0ff98736376b P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.777190 20804 tablet_replica.cc:333] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415: stopping tablet replica
I20251023 02:03:41.777729 20804 raft_consensus.cc:2243] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 4 LEADER]: Raft consensus shutting down.
I20251023 02:03:41.778405 20804 raft_consensus.cc:2272] T 49e9de91bc8041e98d22df0219352ba7 P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.779909 20804 tablet_replica.cc:333] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415: stopping tablet replica
I20251023 02:03:41.780323 20804 raft_consensus.cc:2243] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 1 LEADER]: Raft consensus shutting down.
I20251023 02:03:41.781054 20804 raft_consensus.cc:2272] T 6609dd61840640d796684211f3164998 P 30052fa1e09b406e97bd508d13cac415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.782687 20804 tablet_replica.cc:333] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415: stopping tablet replica
I20251023 02:03:41.783142 20804 raft_consensus.cc:2243] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 2 LEADER]: Raft consensus shutting down.
I20251023 02:03:41.783846 20804 raft_consensus.cc:2272] T 6c36810008544d7fa5c781cae7b1d884 P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.785426 20804 tablet_replica.cc:333] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415: stopping tablet replica
I20251023 02:03:41.785924 20804 raft_consensus.cc:2243] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.786341 20804 raft_consensus.cc:2272] T 676f88b0f9984cba93d7407ac7d88bd3 P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.787896 20804 tablet_replica.cc:333] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415: stopping tablet replica
I20251023 02:03:41.788348 20804 raft_consensus.cc:2243] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.788717 20804 raft_consensus.cc:2272] T d54367d776794a079ca3bf28b8194efc P 30052fa1e09b406e97bd508d13cac415 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.790457 20804 tablet_replica.cc:333] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415: stopping tablet replica
I20251023 02:03:41.790884 20804 raft_consensus.cc:2243] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.791272 20804 raft_consensus.cc:2272] T be886e390b59440393859aef4d6e61f5 P 30052fa1e09b406e97bd508d13cac415 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.792788 20804 tablet_replica.cc:333] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415: stopping tablet replica
I20251023 02:03:41.793191 20804 raft_consensus.cc:2243] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.793538 20804 raft_consensus.cc:2272] T fb614549d596431bbedda15b128caf3e P 30052fa1e09b406e97bd508d13cac415 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.795127 20804 tablet_replica.cc:333] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415: stopping tablet replica
I20251023 02:03:41.795539 20804 raft_consensus.cc:2243] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.795925 20804 raft_consensus.cc:2272] T 69c64628ec8443168d18dbd1f66cc92e P 30052fa1e09b406e97bd508d13cac415 [term 4 FOLLOWER]: Raft consensus is shut down!
W20251023 02:03:41.814435 21052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.81.2:44407: connect: Connection refused (error 111)
W20251023 02:03:41.817296 21052 consensus_peers.cc:597] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 -> Peer 30052fa1e09b406e97bd508d13cac415 (127.20.81.2:44407): Couldn't send request to peer 30052fa1e09b406e97bd508d13cac415. Status: Network error: Client connection negotiation failed: client connection to 127.20.81.2:44407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251023 02:03:41.823220 20804 tablet_server.cc:195] TabletServer@127.20.81.2:0 shutdown complete.
W20251023 02:03:41.831074 21052 consensus_peers.cc:597] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 -> Peer 0130a1960e0740469b5291134a4e78a3 (127.20.81.1:35337): Couldn't send request to peer 0130a1960e0740469b5291134a4e78a3. Status: Network error: Client connection negotiation failed: client connection to 127.20.81.1:35337: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251023 02:03:41.841099 20804 tablet_server.cc:178] TabletServer@127.20.81.3:0 shutting down...
I20251023 02:03:41.863355 20804 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251023 02:03:41.863998 20804 tablet_replica.cc:333] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012: stopping tablet replica
I20251023 02:03:41.864558 20804 raft_consensus.cc:2243] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.865017 20804 raft_consensus.cc:2272] T 69c64628ec8443168d18dbd1f66cc92e P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.866756 20804 tablet_replica.cc:333] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012: stopping tablet replica
I20251023 02:03:41.867274 20804 raft_consensus.cc:2243] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 3 LEADER]: Raft consensus shutting down.
I20251023 02:03:41.868095 20804 raft_consensus.cc:2272] T fb614549d596431bbedda15b128caf3e P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.869625 20804 tablet_replica.cc:333] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012: stopping tablet replica
I20251023 02:03:41.870091 20804 raft_consensus.cc:2243] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.870446 20804 raft_consensus.cc:2272] T 6609dd61840640d796684211f3164998 P 1796d6cb245b41a29d0b808df3338012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.871922 20804 tablet_replica.cc:333] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012: stopping tablet replica
I20251023 02:03:41.872360 20804 raft_consensus.cc:2243] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.872740 20804 raft_consensus.cc:2272] T 49e9de91bc8041e98d22df0219352ba7 P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.874269 20804 tablet_replica.cc:333] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012: stopping tablet replica
I20251023 02:03:41.874706 20804 raft_consensus.cc:2243] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.875084 20804 raft_consensus.cc:2272] T 648a6aac62494cdda1eb0ff98736376b P 1796d6cb245b41a29d0b808df3338012 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.876564 20804 tablet_replica.cc:333] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012: stopping tablet replica
I20251023 02:03:41.876992 20804 raft_consensus.cc:2243] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 2 LEADER]: Raft consensus shutting down.
I20251023 02:03:41.877804 20804 raft_consensus.cc:2272] T d54367d776794a079ca3bf28b8194efc P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.879316 20804 tablet_replica.cc:333] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012: stopping tablet replica
I20251023 02:03:41.879745 20804 raft_consensus.cc:2243] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.880110 20804 raft_consensus.cc:2272] T be886e390b59440393859aef4d6e61f5 P 1796d6cb245b41a29d0b808df3338012 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.881616 20804 tablet_replica.cc:333] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012: stopping tablet replica
I20251023 02:03:41.882069 20804 raft_consensus.cc:2243] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 4 LEADER]: Raft consensus shutting down.
I20251023 02:03:41.882791 20804 raft_consensus.cc:2272] T 676f88b0f9984cba93d7407ac7d88bd3 P 1796d6cb245b41a29d0b808df3338012 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:41.884383 20804 tablet_replica.cc:333] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012: stopping tablet replica
I20251023 02:03:41.884800 20804 raft_consensus.cc:2243] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 02:03:41.885179 20804 raft_consensus.cc:2272] T 6c36810008544d7fa5c781cae7b1d884 P 1796d6cb245b41a29d0b808df3338012 [term 2 FOLLOWER]: Raft consensus is shut down!
W20251023 02:03:42.500769 21109 debug-util.cc:398] Leaking SignalData structure 0x7b080016d6c0 after lost signal to thread 20807
W20251023 02:03:42.501544 21109 debug-util.cc:398] Leaking SignalData structure 0x7b080016dda0 after lost signal to thread 20870
W20251023 02:03:42.910106 20804 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 21109)
W20251023 02:03:43.910523 20804 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 21109)
I20251023 02:03:44.842250 20804 tablet_server.cc:195] TabletServer@127.20.81.3:0 shutdown complete.
I20251023 02:03:44.859086 20804 master.cc:561] Master@127.20.81.62:35173 shutting down...
I20251023 02:03:45.048081 20804 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [term 1 LEADER]: Raft consensus shutting down.
I20251023 02:03:45.048692 20804 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:03:45.049027 20804 tablet_replica.cc:333] T 00000000000000000000000000000000 P c82c7a6ce1a14fe897847751021e80b6: stopping tablet replica
I20251023 02:03:45.067773 20804 master.cc:583] Master@127.20.81.62:35173 shutdown complete.
0130a1960e0740469b5291134a4e78a3 4
1796d6cb245b41a29d0b808df3338012 1
30052fa1e09b406e97bd508d13cac415 4
0130a1960e0740469b5291134a4e78a3 3
1796d6cb245b41a29d0b808df3338012 3
30052fa1e09b406e97bd508d13cac415 3
0130a1960e0740469b5291134a4e78a3 0
1796d6cb245b41a29d0b808df3338012 1
30052fa1e09b406e97bd508d13cac415 8
0130a1960e0740469b5291134a4e78a3 3
1796d6cb245b41a29d0b808df3338012 3
30052fa1e09b406e97bd508d13cac415 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (34227 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20251023 02:03:45.107370 20804 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.81.62:43955
I20251023 02:03:45.108502 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:03:45.114311 21335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:45.114681 21336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:45.115556 21338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:45.115778 20804 server_base.cc:1047] running on GCE node
I20251023 02:03:45.116695 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251023 02:03:45.116902 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:03:45.117019 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761185025117006 us; error 0 us; skew 500 ppm
I20251023 02:03:45.117491 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:03:45.120436 20804 webserver.cc:492] Webserver started at http://127.20.81.62:37317/ using document root <none> and password file <none>
I20251023 02:03:45.120891 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:03:45.121088 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:03:45.121344 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:03:45.122495 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/master-0-root/instance:
uuid: "decddcafd17e486a832f0e000273ccde"
format_stamp: "Formatted at 2025-10-23 02:03:45 on dist-test-slave-6q7l"
I20251023 02:03:45.126675 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20251023 02:03:45.129752 21343 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:45.130420 20804 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251023 02:03:45.130679 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/master-0-root
uuid: "decddcafd17e486a832f0e000273ccde"
format_stamp: "Formatted at 2025-10-23 02:03:45 on dist-test-slave-6q7l"
I20251023 02:03:45.130944 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:03:45.148975 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:03:45.150205 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:03:45.187202 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.62:43955
I20251023 02:03:45.187299 21394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.62:43955 every 8 connection(s)
I20251023 02:03:45.190928 21395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.200160 21395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde: Bootstrap starting.
I20251023 02:03:45.204571 21395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.208444 21395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde: No bootstrap required, opened a new log
I20251023 02:03:45.210408 21395 raft_consensus.cc:359] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decddcafd17e486a832f0e000273ccde" member_type: VOTER }
I20251023 02:03:45.210824 21395 raft_consensus.cc:385] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:45.211061 21395 raft_consensus.cc:740] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decddcafd17e486a832f0e000273ccde, State: Initialized, Role: FOLLOWER
I20251023 02:03:45.211612 21395 consensus_queue.cc:260] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decddcafd17e486a832f0e000273ccde" member_type: VOTER }
I20251023 02:03:45.212056 21395 raft_consensus.cc:399] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 02:03:45.212302 21395 raft_consensus.cc:493] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 02:03:45.212567 21395 raft_consensus.cc:3060] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:45.216236 21395 raft_consensus.cc:515] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decddcafd17e486a832f0e000273ccde" member_type: VOTER }
I20251023 02:03:45.216686 21395 leader_election.cc:304] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: decddcafd17e486a832f0e000273ccde; no voters:
I20251023 02:03:45.217777 21395 leader_election.cc:290] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 02:03:45.218112 21398 raft_consensus.cc:2804] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:45.219372 21398 raft_consensus.cc:697] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [term 1 LEADER]: Becoming Leader. State: Replica: decddcafd17e486a832f0e000273ccde, State: Running, Role: LEADER
I20251023 02:03:45.219900 21398 consensus_queue.cc:237] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decddcafd17e486a832f0e000273ccde" member_type: VOTER }
I20251023 02:03:45.220536 21395 sys_catalog.cc:565] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [sys.catalog]: configured and running, proceeding with master startup.
I20251023 02:03:45.224748 21399 sys_catalog.cc:455] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "decddcafd17e486a832f0e000273ccde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decddcafd17e486a832f0e000273ccde" member_type: VOTER } }
I20251023 02:03:45.225351 21400 sys_catalog.cc:455] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [sys.catalog]: SysCatalogTable state changed. Reason: New leader decddcafd17e486a832f0e000273ccde. Latest consensus state: current_term: 1 leader_uuid: "decddcafd17e486a832f0e000273ccde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decddcafd17e486a832f0e000273ccde" member_type: VOTER } }
I20251023 02:03:45.225567 21399 sys_catalog.cc:458] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [sys.catalog]: This master's current role is: LEADER
I20251023 02:03:45.225986 21400 sys_catalog.cc:458] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [sys.catalog]: This master's current role is: LEADER
I20251023 02:03:45.227527 21406 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 02:03:45.232440 21406 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 02:03:45.235621 20804 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251023 02:03:45.240332 21406 catalog_manager.cc:1357] Generated new cluster ID: a54031287935429a8fc327b63c18a273
I20251023 02:03:45.240556 21406 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 02:03:45.262032 21406 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 02:03:45.263090 21406 catalog_manager.cc:1514] Loading token signing keys...
I20251023 02:03:45.273428 21406 catalog_manager.cc:6022] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde: Generated new TSK 0
I20251023 02:03:45.273952 21406 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 02:03:45.301812 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:03:45.307433 21416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:45.308620 21417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:45.310137 20804 server_base.cc:1047] running on GCE node
W20251023 02:03:45.310182 21419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:45.311369 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251023 02:03:45.311571 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:03:45.311769 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761185025311751 us; error 0 us; skew 500 ppm
I20251023 02:03:45.312458 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:03:45.315018 20804 webserver.cc:492] Webserver started at http://127.20.81.1:40871/ using document root <none> and password file <none>
I20251023 02:03:45.315474 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:03:45.315645 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:03:45.315872 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:03:45.316967 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-0-root/instance:
uuid: "5889ab603ce642848cb8981da16c4b6c"
format_stamp: "Formatted at 2025-10-23 02:03:45 on dist-test-slave-6q7l"
I20251023 02:03:45.321141 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20251023 02:03:45.324379 21424 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:45.325052 20804 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251023 02:03:45.325307 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-0-root
uuid: "5889ab603ce642848cb8981da16c4b6c"
format_stamp: "Formatted at 2025-10-23 02:03:45 on dist-test-slave-6q7l"
I20251023 02:03:45.325567 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:03:45.334830 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:03:45.335907 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:03:45.337230 20804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 02:03:45.339335 20804 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 02:03:45.339511 20804 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:45.339725 20804 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 02:03:45.339879 20804 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:45.378077 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.1:38357
I20251023 02:03:45.378221 21486 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.1:38357 every 8 connection(s)
I20251023 02:03:45.382540 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:03:45.390583 21491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:45.390740 21492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:45.393684 21494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:45.394420 20804 server_base.cc:1047] running on GCE node
I20251023 02:03:45.395972 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251023 02:03:45.396164 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:03:45.396170 21487 heartbeater.cc:344] Connected to a master server at 127.20.81.62:43955
I20251023 02:03:45.396355 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761185025396340 us; error 0 us; skew 500 ppm
I20251023 02:03:45.396586 21487 heartbeater.cc:461] Registering TS with master...
I20251023 02:03:45.396962 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:03:45.397293 21487 heartbeater.cc:507] Master 127.20.81.62:43955 requested a full tablet report, sending...
I20251023 02:03:45.399442 21360 ts_manager.cc:194] Registered new tserver with Master: 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357)
I20251023 02:03:45.399904 20804 webserver.cc:492] Webserver started at http://127.20.81.2:37735/ using document root <none> and password file <none>
I20251023 02:03:45.400555 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:03:45.400797 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:03:45.401142 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:03:45.401885 21360 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35492
I20251023 02:03:45.402735 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-1-root/instance:
uuid: "0fae5335287b4c97a699ab0718d0b886"
format_stamp: "Formatted at 2025-10-23 02:03:45 on dist-test-slave-6q7l"
I20251023 02:03:45.407351 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251023 02:03:45.410508 21499 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:45.411165 20804 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251023 02:03:45.411435 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-1-root
uuid: "0fae5335287b4c97a699ab0718d0b886"
format_stamp: "Formatted at 2025-10-23 02:03:45 on dist-test-slave-6q7l"
I20251023 02:03:45.411681 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:03:45.424952 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:03:45.426122 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:03:45.427615 20804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 02:03:45.429745 20804 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 02:03:45.429941 20804 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:45.430181 20804 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 02:03:45.430387 20804 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:45.468617 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.2:34509
I20251023 02:03:45.468719 21561 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.2:34509 every 8 connection(s)
I20251023 02:03:45.473294 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:03:45.481858 21565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:03:45.483139 21566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:45.484434 20804 server_base.cc:1047] running on GCE node
W20251023 02:03:45.485168 21568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:03:45.485307 21562 heartbeater.cc:344] Connected to a master server at 127.20.81.62:43955
I20251023 02:03:45.485744 21562 heartbeater.cc:461] Registering TS with master...
I20251023 02:03:45.486121 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251023 02:03:45.486372 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:03:45.486451 21562 heartbeater.cc:507] Master 127.20.81.62:43955 requested a full tablet report, sending...
I20251023 02:03:45.486534 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761185025486520 us; error 0 us; skew 500 ppm
I20251023 02:03:45.487185 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:03:45.488456 21360 ts_manager.cc:194] Registered new tserver with Master: 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:45.489684 20804 webserver.cc:492] Webserver started at http://127.20.81.3:41201/ using document root <none> and password file <none>
I20251023 02:03:45.489885 21360 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35498
I20251023 02:03:45.490188 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:03:45.490430 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:03:45.490705 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:03:45.491825 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-2-root/instance:
uuid: "b8bd85025a464734b2619d774a2cbbe7"
format_stamp: "Formatted at 2025-10-23 02:03:45 on dist-test-slave-6q7l"
I20251023 02:03:45.496039 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20251023 02:03:45.499044 21573 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:45.499688 20804 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251023 02:03:45.499979 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-2-root
uuid: "b8bd85025a464734b2619d774a2cbbe7"
format_stamp: "Formatted at 2025-10-23 02:03:45 on dist-test-slave-6q7l"
I20251023 02:03:45.500245 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761184990718903-20804-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:03:45.513062 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:03:45.514217 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:03:45.515580 20804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 02:03:45.517645 20804 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 02:03:45.517845 20804 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:45.518069 20804 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 02:03:45.518220 20804 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 02:03:45.556030 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.3:35189
I20251023 02:03:45.556123 21635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.3:35189 every 8 connection(s)
I20251023 02:03:45.568739 21636 heartbeater.cc:344] Connected to a master server at 127.20.81.62:43955
I20251023 02:03:45.569057 21636 heartbeater.cc:461] Registering TS with master...
I20251023 02:03:45.569707 21636 heartbeater.cc:507] Master 127.20.81.62:43955 requested a full tablet report, sending...
I20251023 02:03:45.571346 21360 ts_manager.cc:194] Registered new tserver with Master: b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:45.571992 20804 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013082234s
I20251023 02:03:45.572561 21360 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35512
I20251023 02:03:45.574450 20804 test_util.cc:276] Using random seed: -673888740
I20251023 02:03:45.594583 21360 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35528:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 02:03:45.597361 21360 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 02:03:45.665149 21601 tablet_service.cc:1505] Processing CreateTablet for tablet 001201838e534f1cbe82a08b450783b1 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251023 02:03:45.666693 21601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 001201838e534f1cbe82a08b450783b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.666826 21599 tablet_service.cc:1505] Processing CreateTablet for tablet bae0f548dc3b4b7d98c2a18278c1df04 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251023 02:03:45.668193 21599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae0f548dc3b4b7d98c2a18278c1df04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.668431 21598 tablet_service.cc:1505] Processing CreateTablet for tablet 08d102136d7e4ba5bff4feed32344f71 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251023 02:03:45.669751 21598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08d102136d7e4ba5bff4feed32344f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.670789 21600 tablet_service.cc:1505] Processing CreateTablet for tablet 2588e3220169455c9dba67a36aa3617a (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251023 02:03:45.672036 21600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2588e3220169455c9dba67a36aa3617a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.674613 21452 tablet_service.cc:1505] Processing CreateTablet for tablet 001201838e534f1cbe82a08b450783b1 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251023 02:03:45.675848 21452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 001201838e534f1cbe82a08b450783b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.675694 21451 tablet_service.cc:1505] Processing CreateTablet for tablet 2588e3220169455c9dba67a36aa3617a (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251023 02:03:45.677034 21451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2588e3220169455c9dba67a36aa3617a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.682770 21450 tablet_service.cc:1505] Processing CreateTablet for tablet bae0f548dc3b4b7d98c2a18278c1df04 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251023 02:03:45.684055 21450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae0f548dc3b4b7d98c2a18278c1df04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.686746 21449 tablet_service.cc:1505] Processing CreateTablet for tablet 08d102136d7e4ba5bff4feed32344f71 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251023 02:03:45.688344 21449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08d102136d7e4ba5bff4feed32344f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.691463 21448 tablet_service.cc:1505] Processing CreateTablet for tablet 507131d09f8b4c548fddfc2c653dc111 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251023 02:03:45.693095 21448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 507131d09f8b4c548fddfc2c653dc111. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.700263 21597 tablet_service.cc:1505] Processing CreateTablet for tablet 507131d09f8b4c548fddfc2c653dc111 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251023 02:03:45.705698 21597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 507131d09f8b4c548fddfc2c653dc111. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.712651 21447 tablet_service.cc:1505] Processing CreateTablet for tablet 19b5f9252c904e698e58220d265f2ee1 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251023 02:03:45.714036 21447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b5f9252c904e698e58220d265f2ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.717511 21524 tablet_service.cc:1505] Processing CreateTablet for tablet 08d102136d7e4ba5bff4feed32344f71 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251023 02:03:45.717905 21527 tablet_service.cc:1505] Processing CreateTablet for tablet 001201838e534f1cbe82a08b450783b1 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251023 02:03:45.718943 21524 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08d102136d7e4ba5bff4feed32344f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.720091 21526 tablet_service.cc:1505] Processing CreateTablet for tablet 2588e3220169455c9dba67a36aa3617a (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251023 02:03:45.721365 21526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2588e3220169455c9dba67a36aa3617a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.722757 21525 tablet_service.cc:1505] Processing CreateTablet for tablet bae0f548dc3b4b7d98c2a18278c1df04 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251023 02:03:45.724890 21523 tablet_service.cc:1505] Processing CreateTablet for tablet 507131d09f8b4c548fddfc2c653dc111 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251023 02:03:45.730015 21527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 001201838e534f1cbe82a08b450783b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.730623 21523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 507131d09f8b4c548fddfc2c653dc111. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.732769 21525 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae0f548dc3b4b7d98c2a18278c1df04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.734213 21651 tablet_bootstrap.cc:492] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7: Bootstrap starting.
I20251023 02:03:45.746239 21652 tablet_bootstrap.cc:492] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c: Bootstrap starting.
I20251023 02:03:45.752866 21652 tablet_bootstrap.cc:654] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.754407 21522 tablet_service.cc:1505] Processing CreateTablet for tablet 19b5f9252c904e698e58220d265f2ee1 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251023 02:03:45.755854 21522 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b5f9252c904e698e58220d265f2ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.755910 21601 tablet_service.cc:1505] Processing CreateTablet for tablet 19b5f9252c904e698e58220d265f2ee1 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251023 02:03:45.757716 21601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b5f9252c904e698e58220d265f2ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.758590 21651 tablet_bootstrap.cc:654] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.774468 21527 tablet_service.cc:1505] Processing CreateTablet for tablet 1bf4495407d9477fa2259679df685567 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251023 02:03:45.775888 21527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf4495407d9477fa2259679df685567. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.780375 21651 tablet_bootstrap.cc:492] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7: No bootstrap required, opened a new log
I20251023 02:03:45.780722 21652 tablet_bootstrap.cc:492] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c: No bootstrap required, opened a new log
I20251023 02:03:45.780851 21651 ts_tablet_manager.cc:1403] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.007s
I20251023 02:03:45.781296 21652 ts_tablet_manager.cc:1403] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.005s
I20251023 02:03:45.788096 21652 raft_consensus.cc:359] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:45.789513 21652 raft_consensus.cc:385] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:45.790508 21447 tablet_service.cc:1505] Processing CreateTablet for tablet 1bf4495407d9477fa2259679df685567 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251023 02:03:45.791198 21652 raft_consensus.cc:740] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Initialized, Role: FOLLOWER
I20251023 02:03:45.791800 21447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf4495407d9477fa2259679df685567. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.794116 21452 tablet_service.cc:1505] Processing CreateTablet for tablet 233ce6142f8040b3a9d0c42c1609ee46 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251023 02:03:45.795324 21452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 233ce6142f8040b3a9d0c42c1609ee46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.795292 21599 tablet_service.cc:1505] Processing CreateTablet for tablet 1bf4495407d9477fa2259679df685567 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251023 02:03:45.796581 21599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf4495407d9477fa2259679df685567. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.806643 21525 tablet_service.cc:1505] Processing CreateTablet for tablet 233ce6142f8040b3a9d0c42c1609ee46 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251023 02:03:45.807919 21525 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 233ce6142f8040b3a9d0c42c1609ee46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.808854 21598 tablet_service.cc:1505] Processing CreateTablet for tablet 233ce6142f8040b3a9d0c42c1609ee46 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251023 02:03:45.810086 21598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 233ce6142f8040b3a9d0c42c1609ee46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.783593 21651 raft_consensus.cc:359] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.814513 21651 raft_consensus.cc:385] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:45.814865 21651 raft_consensus.cc:740] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Initialized, Role: FOLLOWER
I20251023 02:03:45.815867 21651 consensus_queue.cc:260] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.792173 21652 consensus_queue.cc:260] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:45.829457 21597 tablet_service.cc:1505] Processing CreateTablet for tablet 6bb31ddd33004f04b9083b40e4dceec4 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251023 02:03:45.829964 21636 heartbeater.cc:499] Master 127.20.81.62:43955 was elected leader, sending a full tablet report...
I20251023 02:03:45.835284 21597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bb31ddd33004f04b9083b40e4dceec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.842299 21450 tablet_service.cc:1505] Processing CreateTablet for tablet 6bb31ddd33004f04b9083b40e4dceec4 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251023 02:03:45.843526 21450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bb31ddd33004f04b9083b40e4dceec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.846096 21656 raft_consensus.cc:493] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:45.846524 21651 ts_tablet_manager.cc:1434] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7: Time spent starting tablet: real 0.065s user 0.007s sys 0.004s
I20251023 02:03:45.846580 21656 raft_consensus.cc:515] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:45.847535 21651 tablet_bootstrap.cc:492] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7: Bootstrap starting.
I20251023 02:03:45.849125 21656 leader_election.cc:290] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:45.850008 21487 heartbeater.cc:499] Master 127.20.81.62:43955 was elected leader, sending a full tablet report...
I20251023 02:03:45.850294 21448 tablet_service.cc:1505] Processing CreateTablet for tablet b481cc2e0b2b45c8af1284b74317286c (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251023 02:03:45.852296 21653 tablet_bootstrap.cc:492] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886: Bootstrap starting.
I20251023 02:03:45.853175 21448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b481cc2e0b2b45c8af1284b74317286c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.858662 21527 tablet_service.cc:1505] Processing CreateTablet for tablet b481cc2e0b2b45c8af1284b74317286c (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251023 02:03:45.826572 21524 tablet_service.cc:1505] Processing CreateTablet for tablet 6bb31ddd33004f04b9083b40e4dceec4 (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251023 02:03:45.861697 21524 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bb31ddd33004f04b9083b40e4dceec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.864022 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "001201838e534f1cbe82a08b450783b1" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:45.867972 21527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b481cc2e0b2b45c8af1284b74317286c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 02:03:45.869153 21425 leader_election.cc:343] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:45.860672 21653 tablet_bootstrap.cc:654] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.891196 21599 tablet_service.cc:1505] Processing CreateTablet for tablet b481cc2e0b2b45c8af1284b74317286c (DEFAULT_TABLE table=test-workload [id=33b38f9b047c4d058898c4a3988984ac]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251023 02:03:45.899333 21653 tablet_bootstrap.cc:492] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886: No bootstrap required, opened a new log
I20251023 02:03:45.899816 21653 ts_tablet_manager.cc:1403] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.006s
I20251023 02:03:45.900714 21651 tablet_bootstrap.cc:654] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.902995 21652 ts_tablet_manager.cc:1434] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c: Time spent starting tablet: real 0.119s user 0.006s sys 0.011s
I20251023 02:03:45.904029 21652 tablet_bootstrap.cc:492] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c: Bootstrap starting.
I20251023 02:03:45.903918 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "001201838e534f1cbe82a08b450783b1" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
W20251023 02:03:45.905965 21427 leader_election.cc:343] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:45.906672 21427 leader_election.cc:304] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c; no voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7
I20251023 02:03:45.907402 21599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b481cc2e0b2b45c8af1284b74317286c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:03:45.907891 21658 raft_consensus.cc:2749] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:03:45.906641 21653 raft_consensus.cc:359] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.910708 21653 raft_consensus.cc:385] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:45.911085 21653 raft_consensus.cc:740] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Initialized, Role: FOLLOWER
I20251023 02:03:45.911782 21653 consensus_queue.cc:260] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.916872 21651 tablet_bootstrap.cc:492] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7: No bootstrap required, opened a new log
I20251023 02:03:45.917246 21651 ts_tablet_manager.cc:1403] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.014s
I20251023 02:03:45.919309 21653 ts_tablet_manager.cc:1434] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886: Time spent starting tablet: real 0.019s user 0.007s sys 0.003s
I20251023 02:03:45.919860 21562 heartbeater.cc:499] Master 127.20.81.62:43955 was elected leader, sending a full tablet report...
I20251023 02:03:45.919922 21652 tablet_bootstrap.cc:654] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.920131 21651 raft_consensus.cc:359] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.920738 21651 raft_consensus.cc:385] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:45.921030 21651 raft_consensus.cc:740] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Initialized, Role: FOLLOWER
I20251023 02:03:45.922432 21653 tablet_bootstrap.cc:492] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886: Bootstrap starting.
I20251023 02:03:45.925575 21652 tablet_bootstrap.cc:492] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c: No bootstrap required, opened a new log
I20251023 02:03:45.926137 21652 ts_tablet_manager.cc:1403] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.003s
I20251023 02:03:45.928881 21652 raft_consensus.cc:359] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.921855 21651 consensus_queue.cc:260] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.929674 21652 raft_consensus.cc:385] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:45.930054 21652 raft_consensus.cc:740] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Initialized, Role: FOLLOWER
I20251023 02:03:45.931000 21653 tablet_bootstrap.cc:654] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.931245 21651 ts_tablet_manager.cc:1434] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7: Time spent starting tablet: real 0.014s user 0.004s sys 0.003s
I20251023 02:03:45.930787 21652 consensus_queue.cc:260] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.932222 21651 tablet_bootstrap.cc:492] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7: Bootstrap starting.
I20251023 02:03:45.934842 21652 ts_tablet_manager.cc:1434] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20251023 02:03:45.935829 21652 tablet_bootstrap.cc:492] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c: Bootstrap starting.
I20251023 02:03:45.939575 21651 tablet_bootstrap.cc:654] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.942279 21652 tablet_bootstrap.cc:654] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.946333 21653 tablet_bootstrap.cc:492] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886: No bootstrap required, opened a new log
I20251023 02:03:45.946727 21653 ts_tablet_manager.cc:1403] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.011s
I20251023 02:03:45.949750 21651 tablet_bootstrap.cc:492] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7: No bootstrap required, opened a new log
I20251023 02:03:45.949541 21653 raft_consensus.cc:359] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:45.950385 21653 raft_consensus.cc:385] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:45.950491 21651 ts_tablet_manager.cc:1403] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.002s
I20251023 02:03:45.950757 21653 raft_consensus.cc:740] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Initialized, Role: FOLLOWER
I20251023 02:03:45.951423 21653 consensus_queue.cc:260] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:45.953685 21653 ts_tablet_manager.cc:1434] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251023 02:03:45.953402 21651 raft_consensus.cc:359] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:45.954262 21651 raft_consensus.cc:385] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:45.954620 21653 tablet_bootstrap.cc:492] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886: Bootstrap starting.
I20251023 02:03:45.954656 21651 raft_consensus.cc:740] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Initialized, Role: FOLLOWER
I20251023 02:03:45.955775 21651 consensus_queue.cc:260] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:45.963325 21653 tablet_bootstrap.cc:654] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.967788 21651 ts_tablet_manager.cc:1434] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20251023 02:03:45.969008 21651 tablet_bootstrap.cc:492] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7: Bootstrap starting.
I20251023 02:03:45.971177 21652 tablet_bootstrap.cc:492] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c: No bootstrap required, opened a new log
I20251023 02:03:45.971729 21652 ts_tablet_manager.cc:1403] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c: Time spent bootstrapping tablet: real 0.036s user 0.017s sys 0.010s
I20251023 02:03:45.974669 21652 raft_consensus.cc:359] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.975153 21651 tablet_bootstrap.cc:654] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.975487 21652 raft_consensus.cc:385] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:45.975858 21652 raft_consensus.cc:740] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Initialized, Role: FOLLOWER
I20251023 02:03:45.976822 21652 consensus_queue.cc:260] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
W20251023 02:03:45.978350 21563 tablet.cc:2378] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 02:03:45.980517 21653 tablet_bootstrap.cc:492] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886: No bootstrap required, opened a new log
I20251023 02:03:45.980901 21653 ts_tablet_manager.cc:1403] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.001s
I20251023 02:03:45.983058 21652 ts_tablet_manager.cc:1434] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20251023 02:03:45.984107 21652 tablet_bootstrap.cc:492] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c: Bootstrap starting.
I20251023 02:03:45.983441 21653 raft_consensus.cc:359] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.984577 21653 raft_consensus.cc:385] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:45.984938 21653 raft_consensus.cc:740] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Initialized, Role: FOLLOWER
I20251023 02:03:45.985796 21653 consensus_queue.cc:260] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.986768 21651 tablet_bootstrap.cc:492] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7: No bootstrap required, opened a new log
I20251023 02:03:45.987249 21651 ts_tablet_manager.cc:1403] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20251023 02:03:45.989806 21651 raft_consensus.cc:359] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.990622 21652 tablet_bootstrap.cc:654] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:45.990653 21651 raft_consensus.cc:385] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:45.991176 21651 raft_consensus.cc:740] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Initialized, Role: FOLLOWER
I20251023 02:03:45.992022 21651 consensus_queue.cc:260] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:45.992995 21653 ts_tablet_manager.cc:1434] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20251023 02:03:45.994371 21653 tablet_bootstrap.cc:492] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886: Bootstrap starting.
I20251023 02:03:45.996908 21651 ts_tablet_manager.cc:1434] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251023 02:03:45.997889 21651 tablet_bootstrap.cc:492] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7: Bootstrap starting.
I20251023 02:03:46.002686 21653 tablet_bootstrap.cc:654] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.004168 21651 tablet_bootstrap.cc:654] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.008199 21652 tablet_bootstrap.cc:492] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c: No bootstrap required, opened a new log
I20251023 02:03:46.008802 21652 ts_tablet_manager.cc:1403] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.016s
I20251023 02:03:46.012303 21652 raft_consensus.cc:359] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.013257 21652 raft_consensus.cc:385] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.013808 21652 raft_consensus.cc:740] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.014673 21652 consensus_queue.cc:260] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.015874 21653 tablet_bootstrap.cc:492] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886: No bootstrap required, opened a new log
I20251023 02:03:46.016350 21653 ts_tablet_manager.cc:1403] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.007s
I20251023 02:03:46.016453 21651 tablet_bootstrap.cc:492] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7: No bootstrap required, opened a new log
I20251023 02:03:46.016985 21651 ts_tablet_manager.cc:1403] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20251023 02:03:46.016997 21652 ts_tablet_manager.cc:1434] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20251023 02:03:46.018529 21652 tablet_bootstrap.cc:492] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c: Bootstrap starting.
I20251023 02:03:46.019101 21653 raft_consensus.cc:359] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.019866 21653 raft_consensus.cc:385] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.019732 21651 raft_consensus.cc:359] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.020226 21653 raft_consensus.cc:740] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.020514 21651 raft_consensus.cc:385] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.020901 21651 raft_consensus.cc:740] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.021075 21653 consensus_queue.cc:260] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.021792 21651 consensus_queue.cc:260] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.025120 21652 tablet_bootstrap.cc:654] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.025614 21653 ts_tablet_manager.cc:1434] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251023 02:03:46.025882 21657 raft_consensus.cc:493] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.026350 21657 raft_consensus.cc:515] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.026921 21653 tablet_bootstrap.cc:492] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886: Bootstrap starting.
I20251023 02:03:46.028944 21657 leader_election.cc:290] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:46.034144 21653 tablet_bootstrap.cc:654] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.041841 21651 ts_tablet_manager.cc:1434] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20251023 02:03:46.042728 21658 raft_consensus.cc:493] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.043223 21658 raft_consensus.cc:515] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.047502 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
I20251023 02:03:46.047976 21652 tablet_bootstrap.cc:492] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c: No bootstrap required, opened a new log
I20251023 02:03:46.048178 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:46.048486 21652 ts_tablet_manager.cc:1403] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.009s
I20251023 02:03:46.048749 21611 raft_consensus.cc:2468] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 0.
W20251023 02:03:46.049611 21427 leader_election.cc:343] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:46.050721 21425 leader_election.cc:304] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters: 0fae5335287b4c97a699ab0718d0b886
I20251023 02:03:46.051345 21652 raft_consensus.cc:359] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.051880 21656 raft_consensus.cc:2804] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.052453 21656 raft_consensus.cc:493] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.052887 21656 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.055104 21653 tablet_bootstrap.cc:492] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886: No bootstrap required, opened a new log
I20251023 02:03:46.055572 21653 ts_tablet_manager.cc:1403] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.004s
I20251023 02:03:46.056196 21651 tablet_bootstrap.cc:492] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7: Bootstrap starting.
I20251023 02:03:46.055562 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c" is_pre_election: true
I20251023 02:03:46.057912 21658 leader_election.cc:290] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.052067 21652 raft_consensus.cc:385] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.058693 21652 raft_consensus.cc:740] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.058844 21653 raft_consensus.cc:359] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.059464 21652 consensus_queue.cc:260] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.061606 21653 raft_consensus.cc:385] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.061929 21653 raft_consensus.cc:740] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.062229 21656 raft_consensus.cc:515] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.062727 21653 consensus_queue.cc:260] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.063800 21462 raft_consensus.cc:2393] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8bd85025a464734b2619d774a2cbbe7 in current term 1: Already voted for candidate 5889ab603ce642848cb8981da16c4b6c in this term.
I20251023 02:03:46.064968 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:46.065300 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:46.065927 21611 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:03:46.066242 21427 leader_election.cc:343] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:46.067706 21652 ts_tablet_manager.cc:1434] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20251023 02:03:46.067723 21653 ts_tablet_manager.cc:1434] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886: Time spent starting tablet: real 0.012s user 0.005s sys 0.003s
I20251023 02:03:46.068709 21653 tablet_bootstrap.cc:492] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886: Bootstrap starting.
I20251023 02:03:46.069630 21656 leader_election.cc:290] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.072057 21651 tablet_bootstrap.cc:654] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.074776 21653 tablet_bootstrap.cc:654] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.068708 21652 tablet_bootstrap.cc:492] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c: Bootstrap starting.
I20251023 02:03:46.076295 21656 raft_consensus.cc:493] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.076756 21656 raft_consensus.cc:515] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.079813 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:46.080678 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
W20251023 02:03:46.081223 21425 leader_election.cc:343] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:46.081437 21537 raft_consensus.cc:2468] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 0.
I20251023 02:03:46.082098 21652 tablet_bootstrap.cc:654] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.082310 21656 leader_election.cc:290] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.083148 21427 leader_election.cc:304] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters: b8bd85025a464734b2619d774a2cbbe7
I20251023 02:03:46.085963 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
W20251023 02:03:46.087450 21576 leader_election.cc:343] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:46.087997 21576 leader_election.cc:304] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8bd85025a464734b2619d774a2cbbe7; no voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c
I20251023 02:03:46.091528 21658 raft_consensus.cc:2804] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.091926 21658 raft_consensus.cc:493] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.092270 21658 raft_consensus.cc:3060] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.095682 21653 tablet_bootstrap.cc:492] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886: No bootstrap required, opened a new log
I20251023 02:03:46.095682 21611 raft_consensus.cc:2468] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 1.
I20251023 02:03:46.096370 21653 ts_tablet_manager.cc:1403] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.005s
I20251023 02:03:46.096648 21657 raft_consensus.cc:2749] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:03:46.099859 21425 leader_election.cc:304] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters: 0fae5335287b4c97a699ab0718d0b886
I20251023 02:03:46.100003 21653 raft_consensus.cc:359] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.100883 21653 raft_consensus.cc:385] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.101215 21653 raft_consensus.cc:740] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.102195 21653 consensus_queue.cc:260] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.105352 21653 ts_tablet_manager.cc:1434] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251023 02:03:46.108114 21651 tablet_bootstrap.cc:492] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7: No bootstrap required, opened a new log
I20251023 02:03:46.108726 21651 ts_tablet_manager.cc:1403] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7: Time spent bootstrapping tablet: real 0.053s user 0.015s sys 0.007s
I20251023 02:03:46.112287 21651 raft_consensus.cc:359] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.113193 21651 raft_consensus.cc:385] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.113570 21651 raft_consensus.cc:740] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.114619 21651 consensus_queue.cc:260] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.117866 21651 ts_tablet_manager.cc:1434] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7: Time spent starting tablet: real 0.009s user 0.011s sys 0.000s
I20251023 02:03:46.117735 21658 raft_consensus.cc:515] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.119179 21651 tablet_bootstrap.cc:492] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7: Bootstrap starting.
I20251023 02:03:46.120435 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:46.121001 21537 raft_consensus.cc:3060] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.122438 21653 tablet_bootstrap.cc:492] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886: Bootstrap starting.
I20251023 02:03:46.128319 21537 raft_consensus.cc:2468] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 1.
I20251023 02:03:46.129403 21427 leader_election.cc:304] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters:
I20251023 02:03:46.131632 21679 raft_consensus.cc:493] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.131737 21656 raft_consensus.cc:2804] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:46.133421 21658 leader_election.cc:290] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.133950 21653 tablet_bootstrap.cc:654] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.134110 21658 raft_consensus.cc:2804] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:46.132094 21679 raft_consensus.cc:515] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.134645 21658 raft_consensus.cc:697] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:46.137408 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
I20251023 02:03:46.137962 21679 leader_election.cc:290] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.138053 21537 raft_consensus.cc:2468] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 0.
I20251023 02:03:46.139356 21427 leader_election.cc:304] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters:
I20251023 02:03:46.140275 21679 raft_consensus.cc:2804] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.140663 21679 raft_consensus.cc:493] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.141000 21679 raft_consensus.cc:3060] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.142076 21656 raft_consensus.cc:697] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:46.143085 21656 consensus_queue.cc:237] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.144817 21651 tablet_bootstrap.cc:654] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.135499 21658 consensus_queue.cc:237] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
W20251023 02:03:46.151084 21425 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:35189, user_credentials={real_user=slave}} blocked reactor thread for 51351.7us
I20251023 02:03:46.152733 21652 tablet_bootstrap.cc:492] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c: No bootstrap required, opened a new log
I20251023 02:03:46.152733 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:46.153398 21652 ts_tablet_manager.cc:1403] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c: Time spent bootstrapping tablet: real 0.085s user 0.025s sys 0.037s
I20251023 02:03:46.154083 21651 tablet_bootstrap.cc:492] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7: No bootstrap required, opened a new log
I20251023 02:03:46.155058 21651 ts_tablet_manager.cc:1403] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.001s
I20251023 02:03:46.155364 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:46.155992 21611 raft_consensus.cc:2468] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 0.
W20251023 02:03:46.156076 21425 leader_election.cc:343] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:46.156486 21652 raft_consensus.cc:359] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.157202 21652 raft_consensus.cc:385] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.157606 21652 raft_consensus.cc:740] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.158392 21652 consensus_queue.cc:260] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.161026 21652 ts_tablet_manager.cc:1434] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251023 02:03:46.160943 21679 raft_consensus.cc:515] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.160815 21651 raft_consensus.cc:359] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.161738 21651 raft_consensus.cc:385] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.163609 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:46.163812 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:46.164242 21537 raft_consensus.cc:3060] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.164294 21611 raft_consensus.cc:3060] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.164572 21653 tablet_bootstrap.cc:492] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886: No bootstrap required, opened a new log
I20251023 02:03:46.165064 21653 ts_tablet_manager.cc:1403] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.007s
I20251023 02:03:46.167691 21653 raft_consensus.cc:359] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.168335 21653 raft_consensus.cc:385] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.168607 21653 raft_consensus.cc:740] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.162084 21651 raft_consensus.cc:740] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.171516 21537 raft_consensus.cc:2468] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 1.
I20251023 02:03:46.171103 21651 consensus_queue.cc:260] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.171638 21611 raft_consensus.cc:2468] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 1.
I20251023 02:03:46.171140 21360 catalog_manager.cc:5649] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 1 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:46.172902 21427 leader_election.cc:304] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters:
I20251023 02:03:46.162084 21652 tablet_bootstrap.cc:492] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c: Bootstrap starting.
I20251023 02:03:46.173105 21360 catalog_manager.cc:5649] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 1 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:46.173714 21658 raft_consensus.cc:2804] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:46.175256 21658 raft_consensus.cc:697] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:46.176288 21658 consensus_queue.cc:237] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.169265 21653 consensus_queue.cc:260] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.179069 21653 ts_tablet_manager.cc:1434] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
I20251023 02:03:46.179996 21653 tablet_bootstrap.cc:492] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886: Bootstrap starting.
I20251023 02:03:46.180585 21652 tablet_bootstrap.cc:654] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.170459 21679 leader_election.cc:290] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.183779 21651 ts_tablet_manager.cc:1434] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7: Time spent starting tablet: real 0.027s user 0.002s sys 0.004s
I20251023 02:03:46.185112 21651 tablet_bootstrap.cc:492] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7: Bootstrap starting.
I20251023 02:03:46.199995 21653 tablet_bootstrap.cc:654] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.201427 21651 tablet_bootstrap.cc:654] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.205178 21652 tablet_bootstrap.cc:492] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c: No bootstrap required, opened a new log
I20251023 02:03:46.205860 21652 ts_tablet_manager.cc:1403] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.021s
I20251023 02:03:46.209393 21652 raft_consensus.cc:359] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.210527 21652 raft_consensus.cc:385] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.210901 21652 raft_consensus.cc:740] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.211763 21652 consensus_queue.cc:260] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.214632 21652 ts_tablet_manager.cc:1434] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20251023 02:03:46.215559 21652 tablet_bootstrap.cc:492] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c: Bootstrap starting.
I20251023 02:03:46.216042 21653 tablet_bootstrap.cc:492] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886: No bootstrap required, opened a new log
I20251023 02:03:46.216467 21653 ts_tablet_manager.cc:1403] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.008s
I20251023 02:03:46.218823 21653 raft_consensus.cc:359] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.219437 21653 raft_consensus.cc:385] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.219746 21653 raft_consensus.cc:740] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.220556 21653 consensus_queue.cc:260] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.221521 21652 tablet_bootstrap.cc:654] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.224371 21651 tablet_bootstrap.cc:492] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7: No bootstrap required, opened a new log
I20251023 02:03:46.224982 21651 ts_tablet_manager.cc:1403] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.007s
I20251023 02:03:46.227183 21653 ts_tablet_manager.cc:1434] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20251023 02:03:46.228327 21653 tablet_bootstrap.cc:492] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886: Bootstrap starting.
I20251023 02:03:46.228359 21651 raft_consensus.cc:359] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.229486 21651 raft_consensus.cc:385] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.229995 21651 raft_consensus.cc:740] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.230758 21651 consensus_queue.cc:260] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.233188 21651 ts_tablet_manager.cc:1434] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20251023 02:03:46.234115 21651 tablet_bootstrap.cc:492] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7: Bootstrap starting.
I20251023 02:03:46.236268 21653 tablet_bootstrap.cc:654] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.238461 21662 raft_consensus.cc:493] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.239321 21651 tablet_bootstrap.cc:654] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.239073 21662 raft_consensus.cc:515] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.242871 21657 raft_consensus.cc:493] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:03:46.243039 21408 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251023 02:03:46.243343 21657 raft_consensus.cc:515] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.245766 21657 leader_election.cc:290] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:46.248530 21662 leader_election.cc:290] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.264276 21657 raft_consensus.cc:493] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.264882 21657 raft_consensus.cc:515] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.267331 21657 leader_election.cc:290] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:46.269547 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bae0f548dc3b4b7d98c2a18278c1df04" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c" is_pre_election: true
I20251023 02:03:46.256785 21658 raft_consensus.cc:493] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.273170 21658 raft_consensus.cc:515] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.277961 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bae0f548dc3b4b7d98c2a18278c1df04" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:46.278760 21611 raft_consensus.cc:2468] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 0.
I20251023 02:03:46.281975 21425 leader_election.cc:304] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:46.279855 21658 leader_election.cc:290] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.286526 21651 tablet_bootstrap.cc:492] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7: No bootstrap required, opened a new log
I20251023 02:03:46.286969 21651 ts_tablet_manager.cc:1403] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.012s
I20251023 02:03:46.287020 21461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c" is_pre_election: true
I20251023 02:03:46.288271 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bae0f548dc3b4b7d98c2a18278c1df04" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
I20251023 02:03:46.288838 21537 raft_consensus.cc:2468] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 0.
I20251023 02:03:46.280817 21462 raft_consensus.cc:2468] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 0.
I20251023 02:03:46.289589 21651 raft_consensus.cc:359] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.290213 21651 raft_consensus.cc:385] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.290483 21651 raft_consensus.cc:740] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.291330 21679 raft_consensus.cc:2804] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.291380 21651 consensus_queue.cc:260] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.292781 21679 raft_consensus.cc:493] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.293277 21679 raft_consensus.cc:3060] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:03:46.298009 21576 leader_election.cc:343] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:03:46.304142 21576 leader_election.cc:304] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:46.309468 21657 raft_consensus.cc:493] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.310036 21657 raft_consensus.cc:515] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:46.312152 21657 leader_election.cc:290] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357)
I20251023 02:03:46.312861 21657 raft_consensus.cc:2804] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.313275 21657 raft_consensus.cc:493] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.313617 21657 raft_consensus.cc:3060] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.337801 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bae0f548dc3b4b7d98c2a18278c1df04" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
I20251023 02:03:46.336393 21662 raft_consensus.cc:493] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.338371 21537 raft_consensus.cc:2468] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 0.
I20251023 02:03:46.338827 21662 raft_consensus.cc:515] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.339447 21536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
I20251023 02:03:46.339720 21651 ts_tablet_manager.cc:1434] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7: Time spent starting tablet: real 0.052s user 0.007s sys 0.004s
I20251023 02:03:46.340016 21536 raft_consensus.cc:2468] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 0.
I20251023 02:03:46.340608 21651 tablet_bootstrap.cc:492] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7: Bootstrap starting.
I20251023 02:03:46.340605 21662 leader_election.cc:290] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.321453 21679 raft_consensus.cc:515] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.341104 21359 catalog_manager.cc:5649] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 1 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:46.343106 21679 leader_election.cc:290] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.344308 21536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bae0f548dc3b4b7d98c2a18278c1df04" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:46.346429 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bae0f548dc3b4b7d98c2a18278c1df04" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:46.346436 21536 raft_consensus.cc:3060] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.347353 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "001201838e534f1cbe82a08b450783b1" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c" is_pre_election: true
I20251023 02:03:46.347950 21462 raft_consensus.cc:2468] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 0.
I20251023 02:03:46.348959 21652 tablet_bootstrap.cc:492] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c: No bootstrap required, opened a new log
I20251023 02:03:46.349383 21652 ts_tablet_manager.cc:1403] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c: Time spent bootstrapping tablet: real 0.134s user 0.046s sys 0.054s
I20251023 02:03:46.349568 21576 leader_election.cc:304] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:46.353904 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "001201838e534f1cbe82a08b450783b1" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
I20251023 02:03:46.354554 21537 raft_consensus.cc:2468] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 0.
I20251023 02:03:46.357159 21536 raft_consensus.cc:2468] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 1.
I20251023 02:03:46.357492 21696 raft_consensus.cc:2804] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.358395 21427 leader_election.cc:304] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters:
I20251023 02:03:46.359000 21576 leader_election.cc:304] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters: 5889ab603ce642848cb8981da16c4b6c
I20251023 02:03:46.357944 21696 raft_consensus.cc:493] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.361796 21696 raft_consensus.cc:3060] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.365191 21697 raft_consensus.cc:2804] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.365720 21697 raft_consensus.cc:493] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.366146 21697 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.364432 21679 raft_consensus.cc:2804] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:46.368772 21696 raft_consensus.cc:515] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:46.358317 21652 raft_consensus.cc:359] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.369717 21652 raft_consensus.cc:385] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.369395 21657 raft_consensus.cc:515] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.368714 21679 raft_consensus.cc:697] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:46.370802 21611 raft_consensus.cc:2393] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5889ab603ce642848cb8981da16c4b6c in current term 1: Already voted for candidate b8bd85025a464734b2619d774a2cbbe7 in this term.
I20251023 02:03:46.370018 21652 raft_consensus.cc:740] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.371753 21679 consensus_queue.cc:237] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.373721 21657 leader_election.cc:290] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:46.370846 21651 tablet_bootstrap.cc:654] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.374447 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bae0f548dc3b4b7d98c2a18278c1df04" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:03:46.375967 21652 consensus_queue.cc:260] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.378019 21536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bae0f548dc3b4b7d98c2a18278c1df04" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:46.378763 21536 raft_consensus.cc:2393] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8bd85025a464734b2619d774a2cbbe7 in current term 1: Already voted for candidate 5889ab603ce642848cb8981da16c4b6c in this term.
I20251023 02:03:46.379586 21658 raft_consensus.cc:493] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.379945 21662 raft_consensus.cc:493] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.380043 21658 raft_consensus.cc:515] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.380398 21662 raft_consensus.cc:515] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.381482 21652 ts_tablet_manager.cc:1434] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c: Time spent starting tablet: real 0.032s user 0.004s sys 0.001s
I20251023 02:03:46.382033 21658 leader_election.cc:290] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.382663 21652 tablet_bootstrap.cc:492] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c: Bootstrap starting.
I20251023 02:03:46.384483 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:46.384624 21576 leader_election.cc:304] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8bd85025a464734b2619d774a2cbbe7; no voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c
I20251023 02:03:46.385397 21536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
I20251023 02:03:46.385962 21536 raft_consensus.cc:2468] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 0.
I20251023 02:03:46.386668 21653 tablet_bootstrap.cc:492] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886: No bootstrap required, opened a new log
I20251023 02:03:46.387099 21653 ts_tablet_manager.cc:1403] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886: Time spent bootstrapping tablet: real 0.159s user 0.013s sys 0.017s
I20251023 02:03:46.387035 21427 leader_election.cc:304] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters:
I20251023 02:03:46.382124 21662 leader_election.cc:290] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.387982 21662 raft_consensus.cc:493] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.388388 21662 raft_consensus.cc:515] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.389760 21657 raft_consensus.cc:2749] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 02:03:46.390208 21662 leader_election.cc:290] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.391748 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bae0f548dc3b4b7d98c2a18278c1df04" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:46.391935 21658 raft_consensus.cc:2804] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.392270 21658 raft_consensus.cc:493] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.392525 21610 raft_consensus.cc:2393] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fae5335287b4c97a699ab0718d0b886 in current term 1: Already voted for candidate b8bd85025a464734b2619d774a2cbbe7 in this term.
I20251023 02:03:46.392652 21658 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.393781 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:46.394377 21610 raft_consensus.cc:2468] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 0.
I20251023 02:03:46.395437 21696 leader_election.cc:290] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357)
I20251023 02:03:46.395969 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bf4495407d9477fa2259679df685567" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:46.396540 21610 raft_consensus.cc:2468] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 0.
I20251023 02:03:46.396925 21652 tablet_bootstrap.cc:654] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.398131 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:46.398818 21610 raft_consensus.cc:2393] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fae5335287b4c97a699ab0718d0b886 in current term 1: Already voted for candidate 5889ab603ce642848cb8981da16c4b6c in this term.
I20251023 02:03:46.399693 21658 raft_consensus.cc:515] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.400513 21500 leader_election.cc:304] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:46.401357 21662 raft_consensus.cc:2804] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.401717 21662 raft_consensus.cc:493] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.402069 21662 raft_consensus.cc:3060] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.405527 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "001201838e534f1cbe82a08b450783b1" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:03:46.405771 21500 leader_election.cc:304] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:46.406306 21462 raft_consensus.cc:3060] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.407814 21653 raft_consensus.cc:359] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.408479 21653 raft_consensus.cc:385] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.408747 21653 raft_consensus.cc:740] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.410104 21701 raft_consensus.cc:2804] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.410497 21701 raft_consensus.cc:493] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.410840 21701 raft_consensus.cc:3060] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.414372 21536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "001201838e534f1cbe82a08b450783b1" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:46.414600 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:46.414937 21536 raft_consensus.cc:3060] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.415068 21537 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.422428 21461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bae0f548dc3b4b7d98c2a18278c1df04" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c" is_pre_election: true
I20251023 02:03:46.424041 21460 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c" is_pre_election: true
I20251023 02:03:46.434047 21460 raft_consensus.cc:2468] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 0.
I20251023 02:03:46.435464 21459 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bf4495407d9477fa2259679df685567" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c" is_pre_election: true
I20251023 02:03:46.436321 21459 raft_consensus.cc:2468] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 0.
I20251023 02:03:46.436746 21653 consensus_queue.cc:260] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.443841 21696 raft_consensus.cc:493] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.444468 21696 raft_consensus.cc:515] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.460597 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:46.461225 21462 raft_consensus.cc:2468] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 1.
I20251023 02:03:46.462405 21658 leader_election.cc:290] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.465287 21652 tablet_bootstrap.cc:492] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c: No bootstrap required, opened a new log
I20251023 02:03:46.465909 21652 ts_tablet_manager.cc:1403] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c: Time spent bootstrapping tablet: real 0.083s user 0.021s sys 0.013s
I20251023 02:03:46.467458 21653 ts_tablet_manager.cc:1434] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886: Time spent starting tablet: real 0.080s user 0.006s sys 0.001s
I20251023 02:03:46.468185 21459 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c" is_pre_election: true
I20251023 02:03:46.469156 21652 raft_consensus.cc:359] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.470045 21652 raft_consensus.cc:385] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.470629 21652 raft_consensus.cc:740] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.471506 21652 consensus_queue.cc:260] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.470050 21502 leader_election.cc:304] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886; no voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7
I20251023 02:03:46.473457 21576 leader_election.cc:304] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:46.474205 21657 raft_consensus.cc:2804] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:46.476434 21652 ts_tablet_manager.cc:1434] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20251023 02:03:46.477532 21652 tablet_bootstrap.cc:492] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c: Bootstrap starting.
I20251023 02:03:46.473037 21359 catalog_manager.cc:5649] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 1 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:46.480827 21696 leader_election.cc:290] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:46.482813 21535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
I20251023 02:03:46.483948 21535 raft_consensus.cc:2393] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8bd85025a464734b2619d774a2cbbe7 in current term 1: Already voted for candidate 5889ab603ce642848cb8981da16c4b6c in this term.
I20251023 02:03:46.485320 21703 raft_consensus.cc:2749] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:03:46.486624 21502 leader_election.cc:304] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886; no voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7
I20251023 02:03:46.487844 21652 tablet_bootstrap.cc:654] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.491133 21697 raft_consensus.cc:515] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.492650 21611 raft_consensus.cc:2393] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5889ab603ce642848cb8981da16c4b6c in current term 1: Already voted for candidate b8bd85025a464734b2619d774a2cbbe7 in this term.
I20251023 02:03:46.496398 21703 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.496268 21662 raft_consensus.cc:515] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.498229 21653 tablet_bootstrap.cc:492] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886: Bootstrap starting.
I20251023 02:03:46.501957 21459 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c" is_pre_election: true
I20251023 02:03:46.503438 21576 leader_election.cc:304] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8bd85025a464734b2619d774a2cbbe7; no voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c
I20251023 02:03:46.503916 21697 leader_election.cc:290] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:46.504451 21697 raft_consensus.cc:3060] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.505914 21459 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:03:46.506659 21459 raft_consensus.cc:2393] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8bd85025a464734b2619d774a2cbbe7 in current term 1: Already voted for candidate 5889ab603ce642848cb8981da16c4b6c in this term.
I20251023 02:03:46.507458 21535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:46.501048 21657 raft_consensus.cc:697] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 LEADER]: Becoming Leader. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Running, Role: LEADER
I20251023 02:03:46.510782 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:46.513134 21611 raft_consensus.cc:3060] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.513365 21576 leader_election.cc:304] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8bd85025a464734b2619d774a2cbbe7; no voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c
I20251023 02:03:46.513835 21652 tablet_bootstrap.cc:492] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c: No bootstrap required, opened a new log
I20251023 02:03:46.514276 21652 ts_tablet_manager.cc:1403] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.013s
I20251023 02:03:46.514137 21657 consensus_queue.cc:237] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:46.509193 21662 leader_election.cc:290] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.514863 21704 raft_consensus.cc:2749] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 02:03:46.517295 21459 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:03:46.518465 21459 raft_consensus.cc:3060] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.518779 21537 raft_consensus.cc:2468] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 1.
I20251023 02:03:46.520144 21651 tablet_bootstrap.cc:492] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7: No bootstrap required, opened a new log
I20251023 02:03:46.520653 21651 ts_tablet_manager.cc:1403] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7: Time spent bootstrapping tablet: real 0.180s user 0.045s sys 0.073s
I20251023 02:03:46.522639 21701 raft_consensus.cc:515] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.523113 21611 raft_consensus.cc:2468] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 1.
I20251023 02:03:46.523877 21651 raft_consensus.cc:359] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.524144 21500 leader_election.cc:304] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:46.524633 21651 raft_consensus.cc:385] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.524973 21651 raft_consensus.cc:740] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.525014 21662 raft_consensus.cc:2804] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:46.525763 21662 raft_consensus.cc:697] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 1 LEADER]: Becoming Leader. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Running, Role: LEADER
I20251023 02:03:46.525921 21427 leader_election.cc:304] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters: b8bd85025a464734b2619d774a2cbbe7
I20251023 02:03:46.525838 21651 consensus_queue.cc:260] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.526952 21658 raft_consensus.cc:2804] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:46.527432 21658 raft_consensus.cc:697] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:46.528206 21658 consensus_queue.cc:237] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.528970 21651 ts_tablet_manager.cc:1434] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20251023 02:03:46.529065 21697 raft_consensus.cc:2749] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:03:46.530059 21459 raft_consensus.cc:2468] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 1.
I20251023 02:03:46.529340 21652 raft_consensus.cc:359] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.530827 21652 raft_consensus.cc:385] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.531090 21652 raft_consensus.cc:740] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.530958 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bf4495407d9477fa2259679df685567" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:03:46.531551 21462 raft_consensus.cc:3060] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.531973 21652 consensus_queue.cc:260] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.532519 21662 consensus_queue.cc:237] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.533823 21652 ts_tablet_manager.cc:1434] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20251023 02:03:46.535148 21701 leader_election.cc:290] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.536192 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bf4495407d9477fa2259679df685567" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:46.536741 21610 raft_consensus.cc:3060] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.538749 21462 raft_consensus.cc:2468] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 1.
I20251023 02:03:46.539865 21502 leader_election.cc:304] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters:
I20251023 02:03:46.542651 21653 tablet_bootstrap.cc:654] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886: Neither blocks nor log segments found. Creating new log.
I20251023 02:03:46.543686 21536 raft_consensus.cc:2468] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 1.
I20251023 02:03:46.544896 21610 raft_consensus.cc:2468] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 1.
I20251023 02:03:46.542651 21703 raft_consensus.cc:2749] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:03:46.546356 21701 raft_consensus.cc:2804] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:46.547211 21701 raft_consensus.cc:697] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 1 LEADER]: Becoming Leader. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Running, Role: LEADER
I20251023 02:03:46.548066 21701 consensus_queue.cc:237] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.550793 21359 catalog_manager.cc:5649] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 1 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:46.562776 21653 tablet_bootstrap.cc:492] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886: No bootstrap required, opened a new log
I20251023 02:03:46.563318 21653 ts_tablet_manager.cc:1403] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.012s
I20251023 02:03:46.563714 21360 catalog_manager.cc:5649] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3). New cstate: current_term: 1 leader_uuid: "b8bd85025a464734b2619d774a2cbbe7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:46.565778 21653 raft_consensus.cc:359] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.566568 21653 raft_consensus.cc:385] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:03:46.565871 21359 catalog_manager.cc:5649] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "0fae5335287b4c97a699ab0718d0b886" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:46.568974 21653 raft_consensus.cc:740] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Initialized, Role: FOLLOWER
I20251023 02:03:46.569733 21653 consensus_queue.cc:260] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.573313 21658 consensus_queue.cc:1048] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:03:46.575956 21653 ts_tablet_manager.cc:1434] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20251023 02:03:46.583400 21359 catalog_manager.cc:5649] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "0fae5335287b4c97a699ab0718d0b886" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:46.593003 21679 consensus_queue.cc:1048] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:46.649152 21679 raft_consensus.cc:493] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.649660 21679 raft_consensus.cc:515] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.651321 21679 leader_election.cc:290] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.652343 21536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bb31ddd33004f04b9083b40e4dceec4" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
I20251023 02:03:46.652518 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bb31ddd33004f04b9083b40e4dceec4" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:46.653043 21536 raft_consensus.cc:2468] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 0.
I20251023 02:03:46.653136 21610 raft_consensus.cc:2468] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 0.
I20251023 02:03:46.653924 21679 raft_consensus.cc:493] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:03:46.653965 21427 leader_election.cc:304] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters:
I20251023 02:03:46.654800 21678 raft_consensus.cc:2804] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.654444 21679 raft_consensus.cc:515] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.655066 21678 raft_consensus.cc:493] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.655361 21678 raft_consensus.cc:3060] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.657130 21679 leader_election.cc:290] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.657377 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b481cc2e0b2b45c8af1284b74317286c" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7" is_pre_election: true
I20251023 02:03:46.657994 21610 raft_consensus.cc:2468] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 0.
I20251023 02:03:46.658571 21536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b481cc2e0b2b45c8af1284b74317286c" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886" is_pre_election: true
I20251023 02:03:46.659057 21536 raft_consensus.cc:2468] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 0.
I20251023 02:03:46.658962 21425 leader_election.cc:304] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:46.659771 21679 raft_consensus.cc:2804] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:03:46.660099 21679 raft_consensus.cc:493] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:03:46.660415 21679 raft_consensus.cc:3060] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.661924 21678 raft_consensus.cc:515] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.664367 21678 leader_election.cc:290] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.664893 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bb31ddd33004f04b9083b40e4dceec4" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:46.665280 21610 raft_consensus.cc:3060] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.665158 21536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bb31ddd33004f04b9083b40e4dceec4" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:46.665774 21536 raft_consensus.cc:3060] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.667702 21679 raft_consensus.cc:515] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.669493 21679 leader_election.cc:290] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:46.670244 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b481cc2e0b2b45c8af1284b74317286c" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:46.669559 21610 raft_consensus.cc:2468] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 1.
I20251023 02:03:46.670648 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b481cc2e0b2b45c8af1284b74317286c" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:46.670727 21537 raft_consensus.cc:3060] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.671396 21611 raft_consensus.cc:3060] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:03:46.671876 21425 leader_election.cc:304] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:46.672557 21536 raft_consensus.cc:2468] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 1.
I20251023 02:03:46.673424 21679 raft_consensus.cc:2804] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:46.673925 21679 raft_consensus.cc:697] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:46.674726 21679 consensus_queue.cc:237] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.676277 21537 raft_consensus.cc:2468] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 1.
I20251023 02:03:46.677353 21427 leader_election.cc:304] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters:
I20251023 02:03:46.677870 21611 raft_consensus.cc:2468] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 1.
I20251023 02:03:46.678428 21678 raft_consensus.cc:2804] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:03:46.679152 21678 raft_consensus.cc:697] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:46.679960 21678 consensus_queue.cc:237] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:46.682731 21359 catalog_manager.cc:5649] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 1 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:46.693495 21359 catalog_manager.cc:5649] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 1 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:46.698355 21678 consensus_queue.cc:1048] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:46.709729 21678 consensus_queue.cc:1048] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:46.713840 21679 consensus_queue.cc:1048] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:46.724845 21678 consensus_queue.cc:1048] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:46.823670 21610 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "001201838e534f1cbe82a08b450783b1"
dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
mode: GRACEFUL
new_leader_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:52640
I20251023 02:03:46.824214 21610 raft_consensus.cc:606] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 LEADER]: Received request to transfer leadership to TS 5889ab603ce642848cb8981da16c4b6c
I20251023 02:03:46.842288 21537 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111"
dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
mode: GRACEFUL
new_leader_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:52314
I20251023 02:03:46.842869 21537 raft_consensus.cc:606] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 1 LEADER]: Received request to transfer leadership to TS 5889ab603ce642848cb8981da16c4b6c
I20251023 02:03:46.853367 21462 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
mode: GRACEFUL
new_leader_uuid: "0fae5335287b4c97a699ab0718d0b886"
from {username='slave'} at 127.0.0.1:42704
I20251023 02:03:46.853914 21462 raft_consensus.cc:606] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Received request to transfer leadership to TS 0fae5335287b4c97a699ab0718d0b886
I20251023 02:03:46.859951 21462 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
mode: GRACEFUL
new_leader_uuid: "0fae5335287b4c97a699ab0718d0b886"
from {username='slave'} at 127.0.0.1:42704
I20251023 02:03:46.860420 21462 raft_consensus.cc:606] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Received request to transfer leadership to TS 0fae5335287b4c97a699ab0718d0b886
I20251023 02:03:46.864334 21462 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6bb31ddd33004f04b9083b40e4dceec4"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
mode: GRACEFUL
new_leader_uuid: "0fae5335287b4c97a699ab0718d0b886"
from {username='slave'} at 127.0.0.1:42704
I20251023 02:03:46.864759 21462 raft_consensus.cc:606] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Received request to transfer leadership to TS 0fae5335287b4c97a699ab0718d0b886
I20251023 02:03:46.868638 21462 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b481cc2e0b2b45c8af1284b74317286c"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
mode: GRACEFUL
new_leader_uuid: "b8bd85025a464734b2619d774a2cbbe7"
from {username='slave'} at 127.0.0.1:42704
I20251023 02:03:46.869050 21462 raft_consensus.cc:606] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Received request to transfer leadership to TS b8bd85025a464734b2619d774a2cbbe7
I20251023 02:03:46.933213 21658 consensus_queue.cc:1048] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:46.938302 21662 consensus_queue.cc:1048] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:46.945555 21658 consensus_queue.cc:1048] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:46.953934 21701 consensus_queue.cc:1048] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:46.996419 21657 consensus_queue.cc:1048] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:47.005568 21704 consensus_queue.cc:1048] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:47.009853 21701 consensus_queue.cc:1048] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:47.014941 21657 raft_consensus.cc:993] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7: : Instructing follower 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:03:47.015419 21657 raft_consensus.cc:1081] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 LEADER]: Signalling peer 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:03:47.016839 21459 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "001201838e534f1cbe82a08b450783b1"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:42684
I20251023 02:03:47.017432 21459 raft_consensus.cc:493] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:47.017390 21662 raft_consensus.cc:993] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886: : Instructing follower 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:03:47.017922 21459 raft_consensus.cc:3060] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.018100 21701 raft_consensus.cc:1081] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 1 LEADER]: Signalling peer 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:03:47.021775 21462 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:42696
I20251023 02:03:47.022292 21462 raft_consensus.cc:493] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:47.022547 21462 raft_consensus.cc:3060] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.024569 21705 consensus_queue.cc:1048] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:47.029356 21462 raft_consensus.cc:515] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.031983 21459 raft_consensus.cc:515] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:47.032792 21658 consensus_queue.cc:1048] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:47.034168 21459 leader_election.cc:290] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 2 election: Requested vote from peers b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:47.035668 21536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "001201838e534f1cbe82a08b450783b1" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:47.036216 21536 raft_consensus.cc:3060] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.037566 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "001201838e534f1cbe82a08b450783b1" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:47.038147 21610 raft_consensus.cc:3055] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:47.038538 21610 raft_consensus.cc:740] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Running, Role: LEADER
I20251023 02:03:47.040275 21462 leader_election.cc:290] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 2 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:47.040861 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:47.041374 21611 raft_consensus.cc:3060] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.041721 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:47.039369 21610 consensus_queue.cc:260] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:47.042258 21537 raft_consensus.cc:3055] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:47.042625 21537 raft_consensus.cc:740] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Running, Role: LEADER
I20251023 02:03:47.042903 21610 raft_consensus.cc:3060] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.043335 21537 consensus_queue.cc:260] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.044507 21537 raft_consensus.cc:3060] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.047276 21610 raft_consensus.cc:2468] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 2.
I20251023 02:03:47.047972 21611 raft_consensus.cc:2468] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 2.
I20251023 02:03:47.049840 21658 raft_consensus.cc:993] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c: : Instructing follower 0fae5335287b4c97a699ab0718d0b886 to start an election
I20251023 02:03:47.050226 21658 raft_consensus.cc:1081] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Signalling peer 0fae5335287b4c97a699ab0718d0b886 to start an election
I20251023 02:03:47.050343 21425 leader_election.cc:304] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:47.051211 21726 raft_consensus.cc:2804] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:47.051262 21537 raft_consensus.cc:2468] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 2.
I20251023 02:03:47.051750 21726 raft_consensus.cc:697] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:47.052599 21726 consensus_queue.cc:237] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.054761 21535 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46"
dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
from {username='slave'} at 127.0.0.1:52296
I20251023 02:03:47.055296 21535 raft_consensus.cc:493] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:47.055629 21535 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.057541 21425 leader_election.cc:304] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:47.058386 21678 raft_consensus.cc:2804] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:47.062152 21536 raft_consensus.cc:2468] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 2.
I20251023 02:03:47.062659 21357 catalog_manager.cc:5649] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 1 to 2, leader changed from 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2) to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 2 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:47.063025 21535 raft_consensus.cc:515] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.065719 21658 consensus_queue.cc:1048] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:47.066224 21678 raft_consensus.cc:697] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:47.066466 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:03:47.067020 21462 raft_consensus.cc:3055] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:47.067441 21462 raft_consensus.cc:740] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:47.067055 21678 consensus_queue.cc:237] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:03:47.068192 21462 consensus_queue.cc:260] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.069078 21462 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.071391 21537 raft_consensus.cc:1240] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Rejecting Update request from peer 5889ab603ce642848cb8981da16c4b6c for earlier term 1. Current term is 2. Ops: []
I20251023 02:03:47.073786 21535 leader_election.cc:290] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 2 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:47.074841 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:47.076094 21462 raft_consensus.cc:2468] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 2.
I20251023 02:03:47.077313 21502 leader_election.cc:304] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters: b8bd85025a464734b2619d774a2cbbe7
I20251023 02:03:47.078073 21701 raft_consensus.cc:2804] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:47.078540 21701 raft_consensus.cc:697] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Becoming Leader. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Running, Role: LEADER
I20251023 02:03:47.079329 21701 consensus_queue.cc:237] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.083673 21357 catalog_manager.cc:5649] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 1 to 2, leader changed from b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3) to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 2 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:47.089196 21678 raft_consensus.cc:993] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c: : Instructing follower 0fae5335287b4c97a699ab0718d0b886 to start an election
I20251023 02:03:47.089645 21736 raft_consensus.cc:1081] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Signalling peer 0fae5335287b4c97a699ab0718d0b886 to start an election
I20251023 02:03:47.090952 21537 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1"
dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
from {username='slave'} at 127.0.0.1:52296
I20251023 02:03:47.091519 21537 raft_consensus.cc:493] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:47.092059 21537 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.093783 21359 catalog_manager.cc:5649] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 reported cstate change: term changed from 1 to 2, leader changed from 5889ab603ce642848cb8981da16c4b6c (127.20.81.1) to 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2). New cstate: current_term: 2 leader_uuid: "0fae5335287b4c97a699ab0718d0b886" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:47.097976 21537 raft_consensus.cc:515] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.100690 21537 leader_election.cc:290] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 2 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:47.101276 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:03:47.101893 21462 raft_consensus.cc:3055] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:47.102228 21462 raft_consensus.cc:740] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:47.103049 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:47.103607 21610 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.103017 21462 consensus_queue.cc:260] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
W20251023 02:03:47.104153 21736 consensus_queue.cc:1175] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b8bd85025a464734b2619d774a2cbbe7" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20251023 02:03:47.105214 21462 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.110250 21610 raft_consensus.cc:2468] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 2.
I20251023 02:03:47.110333 21462 raft_consensus.cc:2468] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 2.
I20251023 02:03:47.111299 21500 leader_election.cc:304] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:47.111936 21701 raft_consensus.cc:2804] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:47.112334 21701 raft_consensus.cc:697] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Becoming Leader. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Running, Role: LEADER
I20251023 02:03:47.113013 21701 consensus_queue.cc:237] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.118721 21357 catalog_manager.cc:5649] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 reported cstate change: term changed from 1 to 2, leader changed from 5889ab603ce642848cb8981da16c4b6c (127.20.81.1) to 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2). New cstate: current_term: 2 leader_uuid: "0fae5335287b4c97a699ab0718d0b886" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:47.179924 21736 consensus_queue.cc:1048] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:47.185979 21678 raft_consensus.cc:993] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c: : Instructing follower 0fae5335287b4c97a699ab0718d0b886 to start an election
I20251023 02:03:47.186456 21679 raft_consensus.cc:1081] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Signalling peer 0fae5335287b4c97a699ab0718d0b886 to start an election
I20251023 02:03:47.187885 21535 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6bb31ddd33004f04b9083b40e4dceec4"
dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
from {username='slave'} at 127.0.0.1:52296
I20251023 02:03:47.188431 21535 raft_consensus.cc:493] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:47.188753 21535 raft_consensus.cc:3060] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.190405 21736 consensus_queue.cc:1048] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:47.194808 21535 raft_consensus.cc:515] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.195892 21537 raft_consensus.cc:1240] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Rejecting Update request from peer 5889ab603ce642848cb8981da16c4b6c for earlier term 1. Current term is 2. Ops: []
I20251023 02:03:47.197427 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bb31ddd33004f04b9083b40e4dceec4" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:03:47.197686 21736 consensus_queue.cc:1059] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:47.199069 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bb31ddd33004f04b9083b40e4dceec4" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:03:47.199617 21462 raft_consensus.cc:3055] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:47.199661 21610 raft_consensus.cc:3060] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.199980 21462 raft_consensus.cc:740] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:47.200665 21462 consensus_queue.cc:260] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.201476 21462 raft_consensus.cc:3060] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.202322 21535 leader_election.cc:290] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 2 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:03:47.205822 21610 raft_consensus.cc:2468] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 2.
I20251023 02:03:47.206422 21736 consensus_queue.cc:1048] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:47.207265 21462 raft_consensus.cc:2468] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 2.
W20251023 02:03:47.207693 21679 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20251023 02:03:47.208655 21500 leader_election.cc:304] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:47.209817 21701 raft_consensus.cc:2804] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:47.210170 21701 raft_consensus.cc:697] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Becoming Leader. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Running, Role: LEADER
I20251023 02:03:47.210903 21701 consensus_queue.cc:237] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.218006 21357 catalog_manager.cc:5649] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 reported cstate change: term changed from 1 to 2, leader changed from 5889ab603ce642848cb8981da16c4b6c (127.20.81.1) to 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2). New cstate: current_term: 2 leader_uuid: "0fae5335287b4c97a699ab0718d0b886" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:03:47.221450 21736 consensus_queue.cc:1048] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:03:47.229053 21678 raft_consensus.cc:993] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c: : Instructing follower b8bd85025a464734b2619d774a2cbbe7 to start an election
I20251023 02:03:47.229488 21736 raft_consensus.cc:1081] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Signalling peer b8bd85025a464734b2619d774a2cbbe7 to start an election
I20251023 02:03:47.230971 21610 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b481cc2e0b2b45c8af1284b74317286c"
dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
from {username='slave'} at 127.0.0.1:52632
I20251023 02:03:47.231396 21610 raft_consensus.cc:493] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:47.231663 21610 raft_consensus.cc:3060] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.236594 21610 raft_consensus.cc:515] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.238304 21610 leader_election.cc:290] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 2 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:47.239001 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b481cc2e0b2b45c8af1284b74317286c" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:03:47.239284 21535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b481cc2e0b2b45c8af1284b74317286c" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:47.239579 21462 raft_consensus.cc:3055] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:47.239799 21535 raft_consensus.cc:3060] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.239914 21462 raft_consensus.cc:740] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:47.240725 21462 consensus_queue.cc:260] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.241931 21462 raft_consensus.cc:3060] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.246758 21535 raft_consensus.cc:2468] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 2.
I20251023 02:03:47.247952 21576 leader_election.cc:304] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:47.248792 21657 raft_consensus.cc:2804] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:47.249275 21657 raft_consensus.cc:697] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Becoming Leader. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Running, Role: LEADER
I20251023 02:03:47.250473 21462 raft_consensus.cc:2468] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 2.
I20251023 02:03:47.250113 21657 consensus_queue.cc:237] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:47.256572 21360 catalog_manager.cc:5649] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 reported cstate change: term changed from 1 to 2, leader changed from 5889ab603ce642848cb8981da16c4b6c (127.20.81.1) to b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3). New cstate: current_term: 2 leader_uuid: "b8bd85025a464734b2619d774a2cbbe7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:47.450443 21535 raft_consensus.cc:1275] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.451629 21736 consensus_queue.cc:1048] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:47.459046 21610 raft_consensus.cc:1275] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.460389 21736 consensus_queue.cc:1048] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:47.490458 21462 raft_consensus.cc:1275] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Refusing update from remote peer 0fae5335287b4c97a699ab0718d0b886: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.491534 21701 consensus_queue.cc:1048] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:47.498589 21610 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:47.505872 21610 raft_consensus.cc:1275] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Refusing update from remote peer 0fae5335287b4c97a699ab0718d0b886: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.507045 21701 consensus_queue.cc:1048] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:47.535755 21535 raft_consensus.cc:1275] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.536804 21736 consensus_queue.cc:1048] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:47.543375 21610 raft_consensus.cc:1275] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.544657 21679 consensus_queue.cc:1048] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:47.627893 21610 raft_consensus.cc:1275] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Refusing update from remote peer 0fae5335287b4c97a699ab0718d0b886: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.629060 21701 consensus_queue.cc:1048] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:47.636495 21462 raft_consensus.cc:1275] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Refusing update from remote peer 0fae5335287b4c97a699ab0718d0b886: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.636924 21535 raft_consensus.cc:1275] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Refusing update from remote peer b8bd85025a464734b2619d774a2cbbe7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.637742 21701 consensus_queue.cc:1048] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:47.639142 21657 consensus_queue.cc:1048] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:47.649806 21462 raft_consensus.cc:1275] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Refusing update from remote peer b8bd85025a464734b2619d774a2cbbe7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.651774 21657 consensus_queue.cc:1048] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:47.653441 21611 raft_consensus.cc:1275] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Refusing update from remote peer 0fae5335287b4c97a699ab0718d0b886: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.657979 21703 consensus_queue.cc:1048] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:47.659919 21462 raft_consensus.cc:1275] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Refusing update from remote peer 0fae5335287b4c97a699ab0718d0b886: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:47.661048 21703 consensus_queue.cc:1048] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:49.896195 20804 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20251023 02:03:49.896786 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:49.916122 21462 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
mode: GRACEFUL
new_leader_uuid: "b8bd85025a464734b2619d774a2cbbe7"
from {username='slave'} at 127.0.0.1:42712
I20251023 02:03:49.916601 21462 raft_consensus.cc:606] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Received request to transfer leadership to TS b8bd85025a464734b2619d774a2cbbe7
I20251023 02:03:49.918835 21462 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
mode: GRACEFUL
new_leader_uuid: "b8bd85025a464734b2619d774a2cbbe7"
from {username='slave'} at 127.0.0.1:42712
I20251023 02:03:49.919253 21462 raft_consensus.cc:606] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Received request to transfer leadership to TS b8bd85025a464734b2619d774a2cbbe7
I20251023 02:03:49.920056 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251023 02:03:49.920408 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:50.325404 21750 raft_consensus.cc:993] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c: : Instructing follower b8bd85025a464734b2619d774a2cbbe7 to start an election
I20251023 02:03:50.325853 21745 raft_consensus.cc:1081] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Signalling peer b8bd85025a464734b2619d774a2cbbe7 to start an election
I20251023 02:03:50.327114 21610 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71"
dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
from {username='slave'} at 127.0.0.1:52632
I20251023 02:03:50.327548 21610 raft_consensus.cc:493] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:50.327786 21610 raft_consensus.cc:3060] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:50.331841 21610 raft_consensus.cc:515] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:50.333021 21610 leader_election.cc:290] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 2 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:50.333782 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:03:50.334103 21535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:50.334247 21462 raft_consensus.cc:3055] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:50.334554 21462 raft_consensus.cc:740] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:50.334564 21535 raft_consensus.cc:3060] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:50.335227 21462 consensus_queue.cc:260] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:50.336089 21462 raft_consensus.cc:3060] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:50.338918 21535 raft_consensus.cc:2468] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 2.
I20251023 02:03:50.339782 21576 leader_election.cc:304] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:50.340351 21746 raft_consensus.cc:2804] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:50.340446 21462 raft_consensus.cc:2468] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 2.
I20251023 02:03:50.341275 21746 raft_consensus.cc:697] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Becoming Leader. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Running, Role: LEADER
I20251023 02:03:50.342171 21746 consensus_queue.cc:237] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:50.349033 21359 catalog_manager.cc:5649] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 reported cstate change: term changed from 1 to 2, leader changed from 5889ab603ce642848cb8981da16c4b6c (127.20.81.1) to b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3). New cstate: current_term: 2 leader_uuid: "b8bd85025a464734b2619d774a2cbbe7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:50.397696 21750 raft_consensus.cc:993] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c: : Instructing follower b8bd85025a464734b2619d774a2cbbe7 to start an election
I20251023 02:03:50.398087 21745 raft_consensus.cc:1081] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Signalling peer b8bd85025a464734b2619d774a2cbbe7 to start an election
I20251023 02:03:50.399291 21610 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a"
dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
from {username='slave'} at 127.0.0.1:52632
I20251023 02:03:50.399730 21610 raft_consensus.cc:493] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:03:50.400005 21610 raft_consensus.cc:3060] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:50.404326 21610 raft_consensus.cc:515] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:50.405524 21610 leader_election.cc:290] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 2 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:03:50.406272 21462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:03:50.406575 21535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:03:50.406733 21462 raft_consensus.cc:3055] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:03:50.407001 21462 raft_consensus.cc:740] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:03:50.407021 21535 raft_consensus.cc:3060] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:50.407696 21462 consensus_queue.cc:260] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:50.408536 21462 raft_consensus.cc:3060] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:03:50.411485 21535 raft_consensus.cc:2468] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 2.
I20251023 02:03:50.412340 21576 leader_election.cc:304] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:03:50.412391 21462 raft_consensus.cc:2468] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 2.
I20251023 02:03:50.413081 21746 raft_consensus.cc:2804] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:03:50.413529 21746 raft_consensus.cc:697] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Becoming Leader. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Running, Role: LEADER
I20251023 02:03:50.414284 21746 consensus_queue.cc:237] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:03:50.421214 21359 catalog_manager.cc:5649] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 reported cstate change: term changed from 1 to 2, leader changed from 5889ab603ce642848cb8981da16c4b6c (127.20.81.1) to b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3). New cstate: current_term: 2 leader_uuid: "b8bd85025a464734b2619d774a2cbbe7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: HEALTHY } } }
I20251023 02:03:50.745333 21462 raft_consensus.cc:1275] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Refusing update from remote peer b8bd85025a464734b2619d774a2cbbe7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:50.746491 21746 consensus_queue.cc:1048] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:50.759094 21535 raft_consensus.cc:1275] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Refusing update from remote peer b8bd85025a464734b2619d774a2cbbe7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:50.762648 21746 consensus_queue.cc:1048] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:50.837673 21535 raft_consensus.cc:1275] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Refusing update from remote peer b8bd85025a464734b2619d774a2cbbe7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:50.838891 21751 consensus_queue.cc:1048] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:50.846760 21462 raft_consensus.cc:1275] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Refusing update from remote peer b8bd85025a464734b2619d774a2cbbe7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:03:50.848145 21746 consensus_queue.cc:1048] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:03:50.921394 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:50.927798 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:50.928381 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:51.929414 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:51.935446 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:51.935817 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:52.936795 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:52.941841 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:52.942163 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:53.942966 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:53.947883 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:53.948220 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:54.948966 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:54.954730 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:54.955101 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:55.955940 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:55.961349 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:55.961717 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:56.962529 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:56.967653 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:56.967998 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:57.968863 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:57.973929 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:57.974296 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:03:58.975183 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:03:58.980357 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:03:58.980697 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:04:00.007025 20804 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20251023 02:04:00.013340 21611 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a"
dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
mode: GRACEFUL
new_leader_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:52640
I20251023 02:04:00.014017 21611 raft_consensus.cc:606] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Received request to transfer leadership to TS 5889ab603ce642848cb8981da16c4b6c
I20251023 02:04:00.020594 21611 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71"
dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
mode: GRACEFUL
new_leader_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:52640
I20251023 02:04:00.021052 21611 raft_consensus.cc:606] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Received request to transfer leadership to TS 5889ab603ce642848cb8981da16c4b6c
I20251023 02:04:00.027019 21535 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1"
dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
mode: GRACEFUL
new_leader_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:52314
I20251023 02:04:00.027441 21535 raft_consensus.cc:606] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Received request to transfer leadership to TS 5889ab603ce642848cb8981da16c4b6c
I20251023 02:04:00.031477 21535 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1bf4495407d9477fa2259679df685567"
dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
mode: GRACEFUL
new_leader_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:52314
I20251023 02:04:00.031888 21535 raft_consensus.cc:606] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 1 LEADER]: Received request to transfer leadership to TS 5889ab603ce642848cb8981da16c4b6c
I20251023 02:04:00.036383 21537 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46"
dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
mode: GRACEFUL
new_leader_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:52314
I20251023 02:04:00.036985 21537 raft_consensus.cc:606] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Received request to transfer leadership to TS 5889ab603ce642848cb8981da16c4b6c
I20251023 02:04:00.102506 21769 raft_consensus.cc:993] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7: : Instructing follower 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:04:00.103041 21770 raft_consensus.cc:1081] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Signalling peer 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:04:00.104300 21459 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:42684
I20251023 02:04:00.104732 21459 raft_consensus.cc:493] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:00.105010 21459 raft_consensus.cc:3060] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.111008 21459 raft_consensus.cc:515] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.112978 21459 leader_election.cc:290] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 3 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:04:00.113627 21535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:04:00.114620 21535 raft_consensus.cc:3060] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.114079 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2588e3220169455c9dba67a36aa3617a" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:04:00.115226 21611 raft_consensus.cc:3055] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:00.115553 21611 raft_consensus.cc:740] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Running, Role: LEADER
I20251023 02:04:00.116672 21611 consensus_queue.cc:260] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.118003 21611 raft_consensus.cc:3060] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.121850 21535 raft_consensus.cc:2468] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 3.
I20251023 02:04:00.122931 21427 leader_election.cc:304] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters:
I20251023 02:04:00.123579 21611 raft_consensus.cc:2468] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 3.
I20251023 02:04:00.123615 21745 raft_consensus.cc:2804] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:00.124749 21745 raft_consensus.cc:697] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:04:00.125620 21745 consensus_queue.cc:237] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.132208 21360 catalog_manager.cc:5649] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 2 to 3, leader changed from b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3) to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 3 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:00.151975 21705 raft_consensus.cc:993] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886: : Instructing follower 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:04:00.152501 21777 raft_consensus.cc:1081] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Signalling peer 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:04:00.153795 21459 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:42696
I20251023 02:04:00.154224 21459 raft_consensus.cc:493] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:00.154500 21459 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.158933 21459 raft_consensus.cc:515] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.160287 21459 leader_election.cc:290] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 3 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:04:00.161051 21535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:04:00.161161 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:04:00.161537 21535 raft_consensus.cc:3055] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:00.161561 21611 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.161835 21535 raft_consensus.cc:740] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Running, Role: LEADER
I20251023 02:04:00.162544 21535 consensus_queue.cc:260] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.163398 21535 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.165628 21611 raft_consensus.cc:2468] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 3.
I20251023 02:04:00.166631 21425 leader_election.cc:304] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:04:00.167347 21771 raft_consensus.cc:2804] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:00.167440 21535 raft_consensus.cc:2468] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 3.
I20251023 02:04:00.167719 21771 raft_consensus.cc:697] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:04:00.168509 21771 consensus_queue.cc:237] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.174609 21360 catalog_manager.cc:5649] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 2 to 3, leader changed from 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2) to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 3 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:00.258342 21769 raft_consensus.cc:993] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7: : Instructing follower 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:04:00.258787 21770 raft_consensus.cc:1081] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Signalling peer 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:04:00.259974 21459 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:42684
I20251023 02:04:00.260392 21459 raft_consensus.cc:493] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:00.260640 21459 raft_consensus.cc:3060] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.265106 21459 raft_consensus.cc:515] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.266681 21459 leader_election.cc:290] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 3 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:04:00.267360 21535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:04:00.267642 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08d102136d7e4ba5bff4feed32344f71" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:04:00.267849 21535 raft_consensus.cc:3060] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.268132 21611 raft_consensus.cc:3055] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:00.268374 21611 raft_consensus.cc:740] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Running, Role: LEADER
I20251023 02:04:00.268971 21611 consensus_queue.cc:260] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.269788 21611 raft_consensus.cc:3060] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.271945 21535 raft_consensus.cc:2468] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 3.
I20251023 02:04:00.272832 21427 leader_election.cc:304] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, 5889ab603ce642848cb8981da16c4b6c; no voters:
I20251023 02:04:00.273527 21771 raft_consensus.cc:2804] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:00.274024 21771 raft_consensus.cc:697] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:04:00.274227 21611 raft_consensus.cc:2468] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 3.
I20251023 02:04:00.274890 21771 consensus_queue.cc:237] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.282205 21360 catalog_manager.cc:5649] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 2 to 3, leader changed from b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3) to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 3 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:00.284186 21705 raft_consensus.cc:993] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886: : Instructing follower 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:04:00.284575 21705 raft_consensus.cc:1081] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 1 LEADER]: Signalling peer 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:04:00.285970 21459 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1bf4495407d9477fa2259679df685567"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:42696
I20251023 02:04:00.286386 21459 raft_consensus.cc:493] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:00.286754 21459 raft_consensus.cc:3060] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:00.292254 21459 raft_consensus.cc:515] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.293568 21459 leader_election.cc:290] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 2 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:04:00.294348 21535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bf4495407d9477fa2259679df685567" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:04:00.294538 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bf4495407d9477fa2259679df685567" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:04:00.294930 21535 raft_consensus.cc:3055] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:00.294972 21611 raft_consensus.cc:3060] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:00.295332 21535 raft_consensus.cc:740] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Running, Role: LEADER
I20251023 02:04:00.296047 21535 consensus_queue.cc:260] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.297194 21535 raft_consensus.cc:3060] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:00.301070 21611 raft_consensus.cc:2468] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 2.
I20251023 02:04:00.302006 21425 leader_election.cc:304] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:04:00.302639 21771 raft_consensus.cc:2804] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:00.302829 21535 raft_consensus.cc:2468] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 2.
I20251023 02:04:00.302996 21771 raft_consensus.cc:697] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:04:00.303851 21771 consensus_queue.cc:237] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.309563 21360 catalog_manager.cc:5649] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 1 to 2, leader changed from 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2) to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 2 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:00.503360 21777 raft_consensus.cc:993] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886: : Instructing follower 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:04:00.503767 21705 raft_consensus.cc:1081] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Signalling peer 5889ab603ce642848cb8981da16c4b6c to start an election
I20251023 02:04:00.504938 21459 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
from {username='slave'} at 127.0.0.1:42696
I20251023 02:04:00.505357 21459 raft_consensus.cc:493] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:00.505640 21459 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.510115 21459 raft_consensus.cc:515] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.511529 21459 leader_election.cc:290] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 3 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:04:00.512403 21611 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:04:00.512432 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "5889ab603ce642848cb8981da16c4b6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:04:00.513033 21611 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.513106 21537 raft_consensus.cc:3055] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:00.513444 21537 raft_consensus.cc:740] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Running, Role: LEADER
I20251023 02:04:00.514154 21537 consensus_queue.cc:260] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.515254 21537 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:00.519388 21611 raft_consensus.cc:2468] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 3.
I20251023 02:04:00.520215 21610 raft_consensus.cc:1240] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Rejecting Update request from peer 0fae5335287b4c97a699ab0718d0b886 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:00.520435 21425 leader_election.cc:304] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5889ab603ce642848cb8981da16c4b6c, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:04:00.521366 21771 raft_consensus.cc:2804] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:00.522132 21771 raft_consensus.cc:697] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Becoming Leader. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:04:00.522326 21537 raft_consensus.cc:2468] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5889ab603ce642848cb8981da16c4b6c in term 3.
I20251023 02:04:00.523231 21771 consensus_queue.cc:237] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:00.530789 21360 catalog_manager.cc:5649] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c reported cstate change: term changed from 2 to 3, leader changed from 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2) to 5889ab603ce642848cb8981da16c4b6c (127.20.81.1). New cstate: current_term: 3 leader_uuid: "5889ab603ce642848cb8981da16c4b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:00.622417 21537 raft_consensus.cc:1275] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:00.623617 21771 consensus_queue.cc:1048] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:00.634330 21610 raft_consensus.cc:1275] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:00.636013 21785 consensus_queue.cc:1048] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:00.691694 21611 raft_consensus.cc:1275] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:00.692716 21785 consensus_queue.cc:1048] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:00.700486 21537 raft_consensus.cc:1275] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:00.702201 21745 consensus_queue.cc:1048] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:00.724999 21537 raft_consensus.cc:1275] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:00.725981 21745 consensus_queue.cc:1048] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:00.733389 21611 raft_consensus.cc:1275] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:00.734714 21785 consensus_queue.cc:1048] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:00.757774 21611 raft_consensus.cc:1275] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:00.758749 21785 consensus_queue.cc:1048] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:00.766124 21537 raft_consensus.cc:1275] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:00.767352 21785 consensus_queue.cc:1048] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:00.969043 21611 raft_consensus.cc:1275] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:00.970088 21785 consensus_queue.cc:1048] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:00.978242 21537 raft_consensus.cc:1275] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Refusing update from remote peer 5889ab603ce642848cb8981da16c4b6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:00.979701 21785 consensus_queue.cc:1048] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:03.068683 20804 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20251023 02:04:03.069403 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:03.076098 21459 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "001201838e534f1cbe82a08b450783b1"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
mode: GRACEFUL
new_leader_uuid: "b8bd85025a464734b2619d774a2cbbe7"
from {username='slave'} at 127.0.0.1:42712
I20251023 02:04:03.076767 21459 raft_consensus.cc:606] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Received request to transfer leadership to TS b8bd85025a464734b2619d774a2cbbe7
I20251023 02:04:03.079048 21459 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
mode: GRACEFUL
new_leader_uuid: "0fae5335287b4c97a699ab0718d0b886"
from {username='slave'} at 127.0.0.1:42712
I20251023 02:04:03.079468 21459 raft_consensus.cc:606] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Received request to transfer leadership to TS 0fae5335287b4c97a699ab0718d0b886
I20251023 02:04:03.081300 21459 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
mode: GRACEFUL
new_leader_uuid: "0fae5335287b4c97a699ab0718d0b886"
from {username='slave'} at 127.0.0.1:42712
I20251023 02:04:03.081710 21459 raft_consensus.cc:606] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Received request to transfer leadership to TS 0fae5335287b4c97a699ab0718d0b886
I20251023 02:04:03.083498 21459 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111"
dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
mode: GRACEFUL
new_leader_uuid: "b8bd85025a464734b2619d774a2cbbe7"
from {username='slave'} at 127.0.0.1:42712
I20251023 02:04:03.083899 21459 raft_consensus.cc:606] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Received request to transfer leadership to TS b8bd85025a464734b2619d774a2cbbe7
I20251023 02:04:03.084748 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251023 02:04:03.085103 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:04:03.129477 21745 raft_consensus.cc:993] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c: : Instructing follower b8bd85025a464734b2619d774a2cbbe7 to start an election
I20251023 02:04:03.130007 21798 raft_consensus.cc:1081] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Signalling peer b8bd85025a464734b2619d774a2cbbe7 to start an election
I20251023 02:04:03.131410 21610 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111"
dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
from {username='slave'} at 127.0.0.1:52632
I20251023 02:04:03.131816 21610 raft_consensus.cc:493] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:03.132069 21610 raft_consensus.cc:3060] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:03.136950 21610 raft_consensus.cc:515] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:03.138747 21610 leader_election.cc:290] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 3 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509)
I20251023 02:04:03.139688 21459 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:04:03.139890 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "507131d09f8b4c548fddfc2c653dc111" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:04:03.140427 21459 raft_consensus.cc:3055] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:03.140610 21537 raft_consensus.cc:3060] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:03.140684 21459 raft_consensus.cc:740] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:04:03.141552 21459 consensus_queue.cc:260] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:03.142866 21459 raft_consensus.cc:3060] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:03.146159 21537 raft_consensus.cc:2468] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 3.
I20251023 02:04:03.147282 21576 leader_election.cc:304] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:04:03.148102 21770 raft_consensus.cc:2804] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:03.148510 21459 raft_consensus.cc:2468] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 3.
I20251023 02:04:03.149127 21770 raft_consensus.cc:697] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 3 LEADER]: Becoming Leader. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Running, Role: LEADER
I20251023 02:04:03.150017 21770 consensus_queue.cc:237] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:03.156325 21359 catalog_manager.cc:5649] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 reported cstate change: term changed from 2 to 3, leader changed from 5889ab603ce642848cb8981da16c4b6c (127.20.81.1) to b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3). New cstate: current_term: 3 leader_uuid: "b8bd85025a464734b2619d774a2cbbe7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:03.261452 21745 raft_consensus.cc:993] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c: : Instructing follower b8bd85025a464734b2619d774a2cbbe7 to start an election
I20251023 02:04:03.261830 21798 raft_consensus.cc:1081] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Signalling peer b8bd85025a464734b2619d774a2cbbe7 to start an election
I20251023 02:04:03.263115 21610 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "001201838e534f1cbe82a08b450783b1"
dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
from {username='slave'} at 127.0.0.1:52632
I20251023 02:04:03.263496 21610 raft_consensus.cc:493] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:03.263710 21610 raft_consensus.cc:3060] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:03.268057 21610 raft_consensus.cc:515] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:04:03.269213 21610 leader_election.cc:290] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 3 election: Requested vote from peers 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2:34509), 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357)
I20251023 02:04:03.270061 21537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "001201838e534f1cbe82a08b450783b1" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
I20251023 02:04:03.270236 21459 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "001201838e534f1cbe82a08b450783b1" candidate_uuid: "b8bd85025a464734b2619d774a2cbbe7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:04:03.270617 21537 raft_consensus.cc:3060] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:03.270707 21459 raft_consensus.cc:3055] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:03.271065 21459 raft_consensus.cc:740] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:04:03.271893 21459 consensus_queue.cc:260] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:04:03.272915 21459 raft_consensus.cc:3060] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:03.275352 21537 raft_consensus.cc:2468] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 3.
I20251023 02:04:03.276175 21576 leader_election.cc:304] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:04:03.276857 21770 raft_consensus.cc:2804] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:03.277226 21770 raft_consensus.cc:697] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 3 LEADER]: Becoming Leader. State: Replica: b8bd85025a464734b2619d774a2cbbe7, State: Running, Role: LEADER
I20251023 02:04:03.278016 21459 raft_consensus.cc:2468] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8bd85025a464734b2619d774a2cbbe7 in term 3.
I20251023 02:04:03.277981 21770 consensus_queue.cc:237] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } }
I20251023 02:04:03.284667 21359 catalog_manager.cc:5649] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 reported cstate change: term changed from 2 to 3, leader changed from 5889ab603ce642848cb8981da16c4b6c (127.20.81.1) to b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3). New cstate: current_term: 3 leader_uuid: "b8bd85025a464734b2619d774a2cbbe7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:03.303961 21745 raft_consensus.cc:993] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c: : Instructing follower 0fae5335287b4c97a699ab0718d0b886 to start an election
I20251023 02:04:03.304358 21745 raft_consensus.cc:1081] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Signalling peer 0fae5335287b4c97a699ab0718d0b886 to start an election
I20251023 02:04:03.305487 21537 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1"
dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
from {username='slave'} at 127.0.0.1:52296
I20251023 02:04:03.305887 21537 raft_consensus.cc:493] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:03.306154 21537 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:03.309947 21537 raft_consensus.cc:515] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:03.311631 21537 leader_election.cc:290] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 4 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:04:03.312366 21459 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:04:03.312655 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19b5f9252c904e698e58220d265f2ee1" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:04:03.312878 21459 raft_consensus.cc:3055] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:03.313235 21459 raft_consensus.cc:740] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:04:03.313254 21610 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:03.314044 21459 consensus_queue.cc:260] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:03.315133 21459 raft_consensus.cc:3060] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:03.319397 21610 raft_consensus.cc:2468] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 4.
I20251023 02:04:03.320600 21500 leader_election.cc:304] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:04:03.321043 21459 raft_consensus.cc:2468] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 4.
I20251023 02:04:03.321527 21705 raft_consensus.cc:2804] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:03.322610 21705 raft_consensus.cc:697] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 4 LEADER]: Becoming Leader. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Running, Role: LEADER
I20251023 02:04:03.323295 21705 consensus_queue.cc:237] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:03.329959 21359 catalog_manager.cc:5649] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 reported cstate change: term changed from 3 to 4, leader changed from 5889ab603ce642848cb8981da16c4b6c (127.20.81.1) to 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "0fae5335287b4c97a699ab0718d0b886" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:03.372035 21745 raft_consensus.cc:993] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c: : Instructing follower 0fae5335287b4c97a699ab0718d0b886 to start an election
I20251023 02:04:03.372409 21798 raft_consensus.cc:1081] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Signalling peer 0fae5335287b4c97a699ab0718d0b886 to start an election
I20251023 02:04:03.373674 21537 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46"
dest_uuid: "0fae5335287b4c97a699ab0718d0b886"
from {username='slave'} at 127.0.0.1:52296
I20251023 02:04:03.374047 21537 raft_consensus.cc:493] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:03.374270 21537 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:03.378337 21537 raft_consensus.cc:515] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:03.379541 21537 leader_election.cc:290] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 4 election: Requested vote from peers 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357), b8bd85025a464734b2619d774a2cbbe7 (127.20.81.3:35189)
I20251023 02:04:03.380306 21459 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5889ab603ce642848cb8981da16c4b6c"
I20251023 02:04:03.380458 21610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "233ce6142f8040b3a9d0c42c1609ee46" candidate_uuid: "0fae5335287b4c97a699ab0718d0b886" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b8bd85025a464734b2619d774a2cbbe7"
I20251023 02:04:03.380728 21459 raft_consensus.cc:3055] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:03.380892 21610 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:03.381011 21459 raft_consensus.cc:740] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5889ab603ce642848cb8981da16c4b6c, State: Running, Role: LEADER
I20251023 02:04:03.381737 21459 consensus_queue.cc:260] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:03.382510 21459 raft_consensus.cc:3060] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:03.385030 21610 raft_consensus.cc:2468] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 4.
I20251023 02:04:03.385867 21500 leader_election.cc:304] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fae5335287b4c97a699ab0718d0b886, b8bd85025a464734b2619d774a2cbbe7; no voters:
I20251023 02:04:03.386301 21459 raft_consensus.cc:2468] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fae5335287b4c97a699ab0718d0b886 in term 4.
I20251023 02:04:03.386471 21705 raft_consensus.cc:2804] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:03.386904 21705 raft_consensus.cc:697] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 4 LEADER]: Becoming Leader. State: Replica: 0fae5335287b4c97a699ab0718d0b886, State: Running, Role: LEADER
I20251023 02:04:03.387634 21705 consensus_queue.cc:237] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } }
I20251023 02:04:03.393860 21359 catalog_manager.cc:5649] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 reported cstate change: term changed from 3 to 4, leader changed from 5889ab603ce642848cb8981da16c4b6c (127.20.81.1) to 0fae5335287b4c97a699ab0718d0b886 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "0fae5335287b4c97a699ab0718d0b886" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:03.606019 21459 raft_consensus.cc:1275] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Refusing update from remote peer b8bd85025a464734b2619d774a2cbbe7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:03.607002 21770 consensus_queue.cc:1048] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:03.616214 21537 raft_consensus.cc:1275] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Refusing update from remote peer b8bd85025a464734b2619d774a2cbbe7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:03.618187 21770 consensus_queue.cc:1048] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:03.783915 21459 raft_consensus.cc:1275] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Refusing update from remote peer b8bd85025a464734b2619d774a2cbbe7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:03.784907 21770 consensus_queue.cc:1048] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:03.792682 21537 raft_consensus.cc:1275] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Refusing update from remote peer b8bd85025a464734b2619d774a2cbbe7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:03.795018 21816 consensus_queue.cc:1048] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fae5335287b4c97a699ab0718d0b886" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:03.795400 21610 raft_consensus.cc:1275] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 4 FOLLOWER]: Refusing update from remote peer 0fae5335287b4c97a699ab0718d0b886: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:04:03.796674 21705 consensus_queue.cc:1048] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:03.804457 21459 raft_consensus.cc:1275] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 4 FOLLOWER]: Refusing update from remote peer 0fae5335287b4c97a699ab0718d0b886: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:04:03.806033 21705 consensus_queue.cc:1048] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:03.893780 21459 raft_consensus.cc:1275] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 4 FOLLOWER]: Refusing update from remote peer 0fae5335287b4c97a699ab0718d0b886: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:04:03.894981 21705 consensus_queue.cc:1048] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5889ab603ce642848cb8981da16c4b6c" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 38357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:03.902410 21610 raft_consensus.cc:1275] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 4 FOLLOWER]: Refusing update from remote peer 0fae5335287b4c97a699ab0718d0b886: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:04:03.903885 21806 consensus_queue.cc:1048] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8bd85025a464734b2619d774a2cbbe7" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:04.086314 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:04.093492 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:04:04.093891 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:04:05.094866 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:05.099877 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:04:05.100217 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:04:06.101135 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:06.107378 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:04:06.107738 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:04:07.108557 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:07.114027 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:04:07.114360 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:04:08.115116 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:08.120230 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:04:08.120602 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:04:09.121531 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:09.127223 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:04:09.127600 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:04:10.128572 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:10.135305 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:04:10.135661 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:04:11.136484 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:11.142037 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:04:11.142410 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:04:12.143405 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:12.149354 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251023 02:04:12.149722 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:04:13.176533 20804 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20251023 02:04:13.177033 20804 tablet_server.cc:178] TabletServer@127.20.81.1:0 shutting down...
I20251023 02:04:13.204507 20804 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251023 02:04:13.205273 20804 tablet_replica.cc:333] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c: stopping tablet replica
I20251023 02:04:13.205958 20804 raft_consensus.cc:2243] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.206537 20804 raft_consensus.cc:2272] T 6bb31ddd33004f04b9083b40e4dceec4 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.208604 20804 tablet_replica.cc:333] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c: stopping tablet replica
I20251023 02:04:13.209069 20804 raft_consensus.cc:2243] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 1 LEADER]: Raft consensus shutting down.
I20251023 02:04:13.209864 20804 raft_consensus.cc:2272] T bae0f548dc3b4b7d98c2a18278c1df04 P 5889ab603ce642848cb8981da16c4b6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.211668 20804 tablet_replica.cc:333] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c: stopping tablet replica
I20251023 02:04:13.212333 20804 raft_consensus.cc:2243] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 4 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.212926 20804 raft_consensus.cc:2272] T 233ce6142f8040b3a9d0c42c1609ee46 P 5889ab603ce642848cb8981da16c4b6c [term 4 FOLLOWER]: Raft consensus is shut down!
W20251023 02:04:13.213281 21502 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
I20251023 02:04:13.215332 20804 tablet_replica.cc:333] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c: stopping tablet replica
I20251023 02:04:13.216176 20804 raft_consensus.cc:2243] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Raft consensus shutting down.
W20251023 02:04:13.216233 21502 consensus_peers.cc:597] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 -> Peer 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357): Couldn't send request to peer 5889ab603ce642848cb8981da16c4b6c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:13.216753 20804 raft_consensus.cc:2272] T 507131d09f8b4c548fddfc2c653dc111 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.218499 20804 tablet_replica.cc:333] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c: stopping tablet replica
I20251023 02:04:13.218924 20804 raft_consensus.cc:2243] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.219285 20804 raft_consensus.cc:2272] T b481cc2e0b2b45c8af1284b74317286c P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.220834 20804 tablet_replica.cc:333] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c: stopping tablet replica
I20251023 02:04:13.221254 20804 raft_consensus.cc:2243] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 2 LEADER]: Raft consensus shutting down.
I20251023 02:04:13.221941 20804 raft_consensus.cc:2272] T 1bf4495407d9477fa2259679df685567 P 5889ab603ce642848cb8981da16c4b6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.223419 20804 tablet_replica.cc:333] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c: stopping tablet replica
I20251023 02:04:13.223872 20804 raft_consensus.cc:2243] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.224223 20804 raft_consensus.cc:2272] T 001201838e534f1cbe82a08b450783b1 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.225932 20804 tablet_replica.cc:333] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c: stopping tablet replica
I20251023 02:04:13.226392 20804 raft_consensus.cc:2243] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Raft consensus shutting down.
I20251023 02:04:13.227098 20804 raft_consensus.cc:2272] T 08d102136d7e4ba5bff4feed32344f71 P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.228585 20804 tablet_replica.cc:333] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c: stopping tablet replica
I20251023 02:04:13.229022 20804 raft_consensus.cc:2243] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 4 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.229370 20804 raft_consensus.cc:2272] T 19b5f9252c904e698e58220d265f2ee1 P 5889ab603ce642848cb8981da16c4b6c [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.231091 20804 tablet_replica.cc:333] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c: stopping tablet replica
I20251023 02:04:13.231482 20804 raft_consensus.cc:2243] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 3 LEADER]: Raft consensus shutting down.
I20251023 02:04:13.232158 20804 raft_consensus.cc:2272] T 2588e3220169455c9dba67a36aa3617a P 5889ab603ce642848cb8981da16c4b6c [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.260000 20804 tablet_server.cc:195] TabletServer@127.20.81.1:0 shutdown complete.
W20251023 02:04:13.274365 21502 consensus_peers.cc:597] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 -> Peer 5889ab603ce642848cb8981da16c4b6c (127.20.81.1:38357): Couldn't send request to peer 5889ab603ce642848cb8981da16c4b6c. Status: Network error: Client connection negotiation failed: client connection to 127.20.81.1:38357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:13.281770 20804 tablet_server.cc:178] TabletServer@127.20.81.2:0 shutting down...
I20251023 02:04:13.305089 20804 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251023 02:04:13.305848 20804 tablet_replica.cc:333] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886: stopping tablet replica
I20251023 02:04:13.306718 20804 raft_consensus.cc:2243] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 2 LEADER]: Raft consensus shutting down.
I20251023 02:04:13.307760 20804 raft_consensus.cc:2272] T 6bb31ddd33004f04b9083b40e4dceec4 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.309535 20804 tablet_replica.cc:333] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886: stopping tablet replica
I20251023 02:04:13.310067 20804 raft_consensus.cc:2243] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 4 LEADER]: Raft consensus shutting down.
I20251023 02:04:13.310840 20804 raft_consensus.cc:2272] T 19b5f9252c904e698e58220d265f2ee1 P 0fae5335287b4c97a699ab0718d0b886 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.312348 20804 tablet_replica.cc:333] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886: stopping tablet replica
I20251023 02:04:13.312799 20804 raft_consensus.cc:2243] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.313222 20804 raft_consensus.cc:2272] T b481cc2e0b2b45c8af1284b74317286c P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.314898 20804 tablet_replica.cc:333] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886: stopping tablet replica
I20251023 02:04:13.315354 20804 raft_consensus.cc:2243] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.315750 20804 raft_consensus.cc:2272] T 1bf4495407d9477fa2259679df685567 P 0fae5335287b4c97a699ab0718d0b886 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.317255 20804 tablet_replica.cc:333] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886: stopping tablet replica
I20251023 02:04:13.317781 20804 raft_consensus.cc:2243] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.318213 20804 raft_consensus.cc:2272] T 001201838e534f1cbe82a08b450783b1 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.319916 20804 tablet_replica.cc:333] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886: stopping tablet replica
I20251023 02:04:13.320349 20804 raft_consensus.cc:2243] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.320748 20804 raft_consensus.cc:2272] T 08d102136d7e4ba5bff4feed32344f71 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.322350 20804 tablet_replica.cc:333] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886: stopping tablet replica
I20251023 02:04:13.322856 20804 raft_consensus.cc:2243] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.323253 20804 raft_consensus.cc:2272] T 2588e3220169455c9dba67a36aa3617a P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.324882 20804 tablet_replica.cc:333] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886: stopping tablet replica
I20251023 02:04:13.325316 20804 raft_consensus.cc:2243] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.325716 20804 raft_consensus.cc:2272] T bae0f548dc3b4b7d98c2a18278c1df04 P 0fae5335287b4c97a699ab0718d0b886 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.327298 20804 tablet_replica.cc:333] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886: stopping tablet replica
I20251023 02:04:13.327744 20804 raft_consensus.cc:2243] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 4 LEADER]: Raft consensus shutting down.
I20251023 02:04:13.328459 20804 raft_consensus.cc:2272] T 233ce6142f8040b3a9d0c42c1609ee46 P 0fae5335287b4c97a699ab0718d0b886 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.330016 20804 tablet_replica.cc:333] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886: stopping tablet replica
I20251023 02:04:13.330430 20804 raft_consensus.cc:2243] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.330782 20804 raft_consensus.cc:2272] T 507131d09f8b4c548fddfc2c653dc111 P 0fae5335287b4c97a699ab0718d0b886 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.357896 20804 tablet_server.cc:195] TabletServer@127.20.81.2:0 shutdown complete.
I20251023 02:04:13.378168 20804 tablet_server.cc:178] TabletServer@127.20.81.3:0 shutting down...
I20251023 02:04:13.402603 20804 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251023 02:04:13.403245 20804 tablet_replica.cc:333] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7: stopping tablet replica
I20251023 02:04:13.403839 20804 raft_consensus.cc:2243] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.404294 20804 raft_consensus.cc:2272] T 6bb31ddd33004f04b9083b40e4dceec4 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.406118 20804 tablet_replica.cc:333] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7: stopping tablet replica
I20251023 02:04:13.406595 20804 raft_consensus.cc:2243] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.407017 20804 raft_consensus.cc:2272] T 2588e3220169455c9dba67a36aa3617a P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.409005 20804 tablet_replica.cc:333] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7: stopping tablet replica
I20251023 02:04:13.409475 20804 raft_consensus.cc:2243] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.409973 20804 raft_consensus.cc:2272] T 19b5f9252c904e698e58220d265f2ee1 P b8bd85025a464734b2619d774a2cbbe7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.411558 20804 tablet_replica.cc:333] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7: stopping tablet replica
I20251023 02:04:13.412004 20804 raft_consensus.cc:2243] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.412407 20804 raft_consensus.cc:2272] T 233ce6142f8040b3a9d0c42c1609ee46 P b8bd85025a464734b2619d774a2cbbe7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.414033 20804 tablet_replica.cc:333] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7: stopping tablet replica
I20251023 02:04:13.414498 20804 raft_consensus.cc:2243] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 3 LEADER]: Raft consensus shutting down.
I20251023 02:04:13.415221 20804 raft_consensus.cc:2272] T 507131d09f8b4c548fddfc2c653dc111 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.416853 20804 tablet_replica.cc:333] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7: stopping tablet replica
I20251023 02:04:13.417301 20804 raft_consensus.cc:2243] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 2 LEADER]: Raft consensus shutting down.
I20251023 02:04:13.418051 20804 raft_consensus.cc:2272] T b481cc2e0b2b45c8af1284b74317286c P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.419551 20804 tablet_replica.cc:333] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7: stopping tablet replica
I20251023 02:04:13.419978 20804 raft_consensus.cc:2243] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.420383 20804 raft_consensus.cc:2272] T 1bf4495407d9477fa2259679df685567 P b8bd85025a464734b2619d774a2cbbe7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.421957 20804 tablet_replica.cc:333] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7: stopping tablet replica
I20251023 02:04:13.422449 20804 raft_consensus.cc:2243] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.422814 20804 raft_consensus.cc:2272] T 08d102136d7e4ba5bff4feed32344f71 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.424366 20804 tablet_replica.cc:333] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7: stopping tablet replica
I20251023 02:04:13.424822 20804 raft_consensus.cc:2243] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 3 LEADER]: Raft consensus shutting down.
I20251023 02:04:13.425487 20804 raft_consensus.cc:2272] T 001201838e534f1cbe82a08b450783b1 P b8bd85025a464734b2619d774a2cbbe7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.427023 20804 tablet_replica.cc:333] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7: stopping tablet replica
I20251023 02:04:13.427553 20804 raft_consensus.cc:2243] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:13.428004 20804 raft_consensus.cc:2272] T bae0f548dc3b4b7d98c2a18278c1df04 P b8bd85025a464734b2619d774a2cbbe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.457041 20804 tablet_server.cc:195] TabletServer@127.20.81.3:0 shutdown complete.
I20251023 02:04:13.475736 20804 master.cc:561] Master@127.20.81.62:43955 shutting down...
I20251023 02:04:13.497411 20804 raft_consensus.cc:2243] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [term 1 LEADER]: Raft consensus shutting down.
I20251023 02:04:13.498042 20804 raft_consensus.cc:2272] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:13.498346 20804 tablet_replica.cc:333] T 00000000000000000000000000000000 P decddcafd17e486a832f0e000273ccde: stopping tablet replica
I20251023 02:04:13.517079 20804 master.cc:583] Master@127.20.81.62:43955 shutdown complete.
0fae5335287b4c97a699ab0718d0b886 4
5889ab603ce642848cb8981da16c4b6c 5
b8bd85025a464734b2619d774a2cbbe7 1
0fae5335287b4c97a699ab0718d0b886 4
5889ab603ce642848cb8981da16c4b6c 3
b8bd85025a464734b2619d774a2cbbe7 3
0fae5335287b4c97a699ab0718d0b886 1
5889ab603ce642848cb8981da16c4b6c 8
b8bd85025a464734b2619d774a2cbbe7 1
0fae5335287b4c97a699ab0718d0b886 3
5889ab603ce642848cb8981da16c4b6c 4
b8bd85025a464734b2619d774a2cbbe7 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28449 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20251023 02:04:13.557144 20804 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.81.62:41229
I20251023 02:04:13.558310 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:04:13.564025 21836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:04:13.564272 21835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:04:13.564582 21838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:04:13.566691 20804 server_base.cc:1047] running on GCE node
I20251023 02:04:13.567523 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251023 02:04:13.567718 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:04:13.567858 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761185053567840 us; error 0 us; skew 500 ppm
I20251023 02:04:13.568375 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:04:13.570668 20804 webserver.cc:492] Webserver started at http://127.20.81.62:37903/ using document root <none> and password file <none>
I20251023 02:04:13.571118 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:04:13.571282 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:04:13.571554 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:04:13.572692 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/master-0-root/instance:
uuid: "aec15eab11a445ed83475b43469d1669"
format_stamp: "Formatted at 2025-10-23 02:04:13 on dist-test-slave-6q7l"
I20251023 02:04:13.576982 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251023 02:04:13.580222 21843 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:13.580920 20804 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251023 02:04:13.581168 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/master-0-root
uuid: "aec15eab11a445ed83475b43469d1669"
format_stamp: "Formatted at 2025-10-23 02:04:13 on dist-test-slave-6q7l"
I20251023 02:04:13.581434 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:04:13.595809 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:04:13.596959 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:04:13.633620 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.62:41229
I20251023 02:04:13.633718 21894 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.62:41229 every 8 connection(s)
I20251023 02:04:13.637296 21895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:13.646745 21895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669: Bootstrap starting.
I20251023 02:04:13.651154 21895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:13.655019 21895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669: No bootstrap required, opened a new log
I20251023 02:04:13.656966 21895 raft_consensus.cc:359] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec15eab11a445ed83475b43469d1669" member_type: VOTER }
I20251023 02:04:13.657367 21895 raft_consensus.cc:385] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:13.657552 21895 raft_consensus.cc:740] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec15eab11a445ed83475b43469d1669, State: Initialized, Role: FOLLOWER
I20251023 02:04:13.658056 21895 consensus_queue.cc:260] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec15eab11a445ed83475b43469d1669" member_type: VOTER }
I20251023 02:04:13.658457 21895 raft_consensus.cc:399] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 02:04:13.658634 21895 raft_consensus.cc:493] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 02:04:13.658834 21895 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:13.662837 21895 raft_consensus.cc:515] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec15eab11a445ed83475b43469d1669" member_type: VOTER }
I20251023 02:04:13.663244 21895 leader_election.cc:304] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aec15eab11a445ed83475b43469d1669; no voters:
I20251023 02:04:13.664287 21895 leader_election.cc:290] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 02:04:13.664719 21898 raft_consensus.cc:2804] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:13.666044 21898 raft_consensus.cc:697] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [term 1 LEADER]: Becoming Leader. State: Replica: aec15eab11a445ed83475b43469d1669, State: Running, Role: LEADER
I20251023 02:04:13.666610 21898 consensus_queue.cc:237] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec15eab11a445ed83475b43469d1669" member_type: VOTER }
I20251023 02:04:13.667181 21895 sys_catalog.cc:565] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 02:04:13.671530 21899 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aec15eab11a445ed83475b43469d1669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec15eab11a445ed83475b43469d1669" member_type: VOTER } }
I20251023 02:04:13.672204 21899 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [sys.catalog]: This master's current role is: LEADER
I20251023 02:04:13.674086 21900 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aec15eab11a445ed83475b43469d1669. Latest consensus state: current_term: 1 leader_uuid: "aec15eab11a445ed83475b43469d1669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec15eab11a445ed83475b43469d1669" member_type: VOTER } }
I20251023 02:04:13.674808 21900 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669 [sys.catalog]: This master's current role is: LEADER
I20251023 02:04:13.681325 21907 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 02:04:13.682196 20804 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251023 02:04:13.686057 21907 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 02:04:13.694157 21907 catalog_manager.cc:1357] Generated new cluster ID: 987645c938ff487da64265e2d3f0ab9e
I20251023 02:04:13.694397 21907 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 02:04:13.735060 21907 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 02:04:13.736248 21907 catalog_manager.cc:1514] Loading token signing keys...
I20251023 02:04:13.752826 21907 catalog_manager.cc:6022] T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669: Generated new TSK 0
I20251023 02:04:13.753364 21907 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 02:04:13.813123 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:04:13.818938 21916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:04:13.820173 21917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:04:13.822054 21919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:04:13.822475 20804 server_base.cc:1047] running on GCE node
I20251023 02:04:13.823354 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251023 02:04:13.823540 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:04:13.823706 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761185053823688 us; error 0 us; skew 500 ppm
I20251023 02:04:13.824208 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:04:13.826534 20804 webserver.cc:492] Webserver started at http://127.20.81.1:33227/ using document root <none> and password file <none>
I20251023 02:04:13.827016 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:04:13.827189 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:04:13.827438 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:04:13.828500 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-0-root/instance:
uuid: "02c53ba06cc3445a8534a84aa76b1c14"
format_stamp: "Formatted at 2025-10-23 02:04:13 on dist-test-slave-6q7l"
I20251023 02:04:13.833010 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251023 02:04:13.836251 21924 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:13.836915 20804 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251023 02:04:13.837181 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-0-root
uuid: "02c53ba06cc3445a8534a84aa76b1c14"
format_stamp: "Formatted at 2025-10-23 02:04:13 on dist-test-slave-6q7l"
I20251023 02:04:13.837455 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:04:13.852687 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:04:13.853986 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:04:13.855440 20804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 02:04:13.857646 20804 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 02:04:13.857837 20804 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:13.858069 20804 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 02:04:13.858233 20804 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:13.897465 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.1:41499
I20251023 02:04:13.897631 21986 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.1:41499 every 8 connection(s)
I20251023 02:04:13.901952 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:04:13.909497 21991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:04:13.909911 21992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:04:13.914470 21994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:04:13.914865 20804 server_base.cc:1047] running on GCE node
I20251023 02:04:13.915393 21987 heartbeater.cc:344] Connected to a master server at 127.20.81.62:41229
I20251023 02:04:13.915823 21987 heartbeater.cc:461] Registering TS with master...
I20251023 02:04:13.916606 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20251023 02:04:13.916700 21987 heartbeater.cc:507] Master 127.20.81.62:41229 requested a full tablet report, sending...
W20251023 02:04:13.916941 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:04:13.917155 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761185053917135 us; error 0 us; skew 500 ppm
I20251023 02:04:13.917834 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:04:13.918905 21860 ts_manager.cc:194] Registered new tserver with Master: 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499)
I20251023 02:04:13.920285 20804 webserver.cc:492] Webserver started at http://127.20.81.2:44751/ using document root <none> and password file <none>
I20251023 02:04:13.920774 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:04:13.920969 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:04:13.920894 21860 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56856
I20251023 02:04:13.921327 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:04:13.922632 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-1-root/instance:
uuid: "818907d63f1f4d84ba267ebe6ee837e0"
format_stamp: "Formatted at 2025-10-23 02:04:13 on dist-test-slave-6q7l"
I20251023 02:04:13.926788 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251023 02:04:13.929860 21999 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:13.930495 20804 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251023 02:04:13.930743 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-1-root
uuid: "818907d63f1f4d84ba267ebe6ee837e0"
format_stamp: "Formatted at 2025-10-23 02:04:13 on dist-test-slave-6q7l"
I20251023 02:04:13.930994 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:04:13.953119 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:04:13.954314 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:04:13.955744 20804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 02:04:13.957899 20804 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 02:04:13.958091 20804 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:13.958326 20804 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 02:04:13.958483 20804 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:13.996402 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.2:40645
I20251023 02:04:13.996495 22061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.2:40645 every 8 connection(s)
I20251023 02:04:14.001130 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:04:14.007285 22065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:04:14.008986 22066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:04:14.011906 20804 server_base.cc:1047] running on GCE node
W20251023 02:04:14.013130 22068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:04:14.014029 22062 heartbeater.cc:344] Connected to a master server at 127.20.81.62:41229
I20251023 02:04:14.014274 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20251023 02:04:14.014434 22062 heartbeater.cc:461] Registering TS with master...
W20251023 02:04:14.014672 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:04:14.014928 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761185054014911 us; error 0 us; skew 500 ppm
I20251023 02:04:14.015309 22062 heartbeater.cc:507] Master 127.20.81.62:41229 requested a full tablet report, sending...
I20251023 02:04:14.015481 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:04:14.017135 21860 ts_manager.cc:194] Registered new tserver with Master: 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:14.018151 20804 webserver.cc:492] Webserver started at http://127.20.81.3:45113/ using document root <none> and password file <none>
I20251023 02:04:14.018620 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:04:14.018590 21860 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56864
I20251023 02:04:14.018929 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:04:14.019227 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:04:14.020437 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-2-root/instance:
uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
format_stamp: "Formatted at 2025-10-23 02:04:14 on dist-test-slave-6q7l"
I20251023 02:04:14.025009 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20251023 02:04:14.028239 22073 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:14.028911 20804 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251023 02:04:14.029170 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-2-root
uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
format_stamp: "Formatted at 2025-10-23 02:04:14 on dist-test-slave-6q7l"
I20251023 02:04:14.029433 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:04:14.042405 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:04:14.043547 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:04:14.044909 20804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 02:04:14.047016 20804 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 02:04:14.047211 20804 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:14.047434 20804 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 02:04:14.047595 20804 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:14.086971 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.3:33899
I20251023 02:04:14.087074 22135 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.3:33899 every 8 connection(s)
I20251023 02:04:14.101907 22136 heartbeater.cc:344] Connected to a master server at 127.20.81.62:41229
I20251023 02:04:14.102257 22136 heartbeater.cc:461] Registering TS with master...
I20251023 02:04:14.103062 22136 heartbeater.cc:507] Master 127.20.81.62:41229 requested a full tablet report, sending...
I20251023 02:04:14.104839 21860 ts_manager.cc:194] Registered new tserver with Master: ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:14.105741 20804 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014240796s
I20251023 02:04:14.106119 21860 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56872
I20251023 02:04:14.108071 20804 test_util.cc:276] Using random seed: -645355116
I20251023 02:04:14.129797 21860 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56882:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 02:04:14.135906 21860 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 02:04:14.340498 21950 tablet_service.cc:1505] Processing CreateTablet for tablet f18823a673d148e892e11c43e6430242 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251023 02:04:14.341212 21951 tablet_service.cc:1505] Processing CreateTablet for tablet c0594e36e0824e13965498e412eee3fb (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251023 02:04:14.342056 21950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18823a673d148e892e11c43e6430242. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.343588 21952 tablet_service.cc:1505] Processing CreateTablet for tablet 24a8a5ca5c0e4c6384f51352dd78f5d0 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251023 02:04:14.344930 21952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24a8a5ca5c0e4c6384f51352dd78f5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.344640 21949 tablet_service.cc:1505] Processing CreateTablet for tablet f09d991e8bee44c98b35a4bd632685bf (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251023 02:04:14.345966 21949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f09d991e8bee44c98b35a4bd632685bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.346805 22027 tablet_service.cc:1505] Processing CreateTablet for tablet 24a8a5ca5c0e4c6384f51352dd78f5d0 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251023 02:04:14.347121 22023 tablet_service.cc:1505] Processing CreateTablet for tablet 365d47bf062b4ebabb6d5f247e4e3e33 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251023 02:04:14.348074 22027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24a8a5ca5c0e4c6384f51352dd78f5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.349296 22024 tablet_service.cc:1505] Processing CreateTablet for tablet f09d991e8bee44c98b35a4bd632685bf (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251023 02:04:14.350636 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f09d991e8bee44c98b35a4bd632685bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.350736 21948 tablet_service.cc:1505] Processing CreateTablet for tablet 365d47bf062b4ebabb6d5f247e4e3e33 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251023 02:04:14.351948 21948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365d47bf062b4ebabb6d5f247e4e3e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.353420 22025 tablet_service.cc:1505] Processing CreateTablet for tablet f18823a673d148e892e11c43e6430242 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251023 02:04:14.354235 22020 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2df37149b145b89207c30e1a0360b0 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251023 02:04:14.354681 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18823a673d148e892e11c43e6430242. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.355505 22020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2df37149b145b89207c30e1a0360b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.356688 22026 tablet_service.cc:1505] Processing CreateTablet for tablet c0594e36e0824e13965498e412eee3fb (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251023 02:04:14.357972 22026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0594e36e0824e13965498e412eee3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.359922 22022 tablet_service.cc:1505] Processing CreateTablet for tablet a48cb01caedc4f359126f3d85087d709 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251023 02:04:14.361290 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48cb01caedc4f359126f3d85087d709. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.343590 21947 tablet_service.cc:1505] Processing CreateTablet for tablet a48cb01caedc4f359126f3d85087d709 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251023 02:04:14.362226 21943 tablet_service.cc:1505] Processing CreateTablet for tablet 4814f8f6136a4c1490329f260ced6c47 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251023 02:04:14.363484 21951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0594e36e0824e13965498e412eee3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.363749 21944 tablet_service.cc:1505] Processing CreateTablet for tablet 57e843357de442f3a618eac9934513ed (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251023 02:04:14.365864 22018 tablet_service.cc:1505] Processing CreateTablet for tablet 4814f8f6136a4c1490329f260ced6c47 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251023 02:04:14.368511 22019 tablet_service.cc:1505] Processing CreateTablet for tablet 57e843357de442f3a618eac9934513ed (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251023 02:04:14.373324 21945 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2df37149b145b89207c30e1a0360b0 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251023 02:04:14.374694 21945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2df37149b145b89207c30e1a0360b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.376154 21943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4814f8f6136a4c1490329f260ced6c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.377892 21947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48cb01caedc4f359126f3d85087d709. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.379528 21946 tablet_service.cc:1505] Processing CreateTablet for tablet d3bb66d66a264e8abdf220531328f12b (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251023 02:04:14.381242 22021 tablet_service.cc:1505] Processing CreateTablet for tablet d3bb66d66a264e8abdf220531328f12b (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251023 02:04:14.381273 21944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e843357de442f3a618eac9934513ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.385892 22018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4814f8f6136a4c1490329f260ced6c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.386514 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365d47bf062b4ebabb6d5f247e4e3e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.389300 22021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3bb66d66a264e8abdf220531328f12b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.393911 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e843357de442f3a618eac9934513ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.401974 21946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3bb66d66a264e8abdf220531328f12b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.422559 22094 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2df37149b145b89207c30e1a0360b0 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251023 02:04:14.423996 22094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2df37149b145b89207c30e1a0360b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.424348 22100 tablet_service.cc:1505] Processing CreateTablet for tablet c0594e36e0824e13965498e412eee3fb (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251023 02:04:14.422700 22101 tablet_service.cc:1505] Processing CreateTablet for tablet 24a8a5ca5c0e4c6384f51352dd78f5d0 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251023 02:04:14.426805 22100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0594e36e0824e13965498e412eee3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.427206 22099 tablet_service.cc:1505] Processing CreateTablet for tablet f18823a673d148e892e11c43e6430242 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251023 02:04:14.428527 22099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18823a673d148e892e11c43e6430242. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.431476 22098 tablet_service.cc:1505] Processing CreateTablet for tablet f09d991e8bee44c98b35a4bd632685bf (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251023 02:04:14.431721 22097 tablet_service.cc:1505] Processing CreateTablet for tablet 365d47bf062b4ebabb6d5f247e4e3e33 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251023 02:04:14.433372 22096 tablet_service.cc:1505] Processing CreateTablet for tablet a48cb01caedc4f359126f3d85087d709 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251023 02:04:14.436805 22095 tablet_service.cc:1505] Processing CreateTablet for tablet d3bb66d66a264e8abdf220531328f12b (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251023 02:04:14.440109 21949 tablet_service.cc:1505] Processing CreateTablet for tablet 12db617f200e4104b5346432288ad306 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251023 02:04:14.441313 21949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12db617f200e4104b5346432288ad306. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.443823 22101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24a8a5ca5c0e4c6384f51352dd78f5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.455169 22093 tablet_service.cc:1505] Processing CreateTablet for tablet 57e843357de442f3a618eac9934513ed (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251023 02:04:14.456912 22093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e843357de442f3a618eac9934513ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.458034 22097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365d47bf062b4ebabb6d5f247e4e3e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.459533 22092 tablet_service.cc:1505] Processing CreateTablet for tablet 4814f8f6136a4c1490329f260ced6c47 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251023 02:04:14.463864 22096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48cb01caedc4f359126f3d85087d709. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.466192 22027 tablet_service.cc:1505] Processing CreateTablet for tablet 12db617f200e4104b5346432288ad306 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251023 02:04:14.467425 22027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12db617f200e4104b5346432288ad306. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.476069 22098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f09d991e8bee44c98b35a4bd632685bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.482695 22095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3bb66d66a264e8abdf220531328f12b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.485150 22092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4814f8f6136a4c1490329f260ced6c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.476059 22026 tablet_service.cc:1505] Processing CreateTablet for tablet 774c010489aa4775bb6d636926900529 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251023 02:04:14.461858 21950 tablet_service.cc:1505] Processing CreateTablet for tablet 774c010489aa4775bb6d636926900529 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251023 02:04:14.496346 22026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774c010489aa4775bb6d636926900529. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.496819 21950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774c010489aa4775bb6d636926900529. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.497591 22023 tablet_service.cc:1505] Processing CreateTablet for tablet 1376a6af91b24e46859bf630618fcec3 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251023 02:04:14.499151 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1376a6af91b24e46859bf630618fcec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.523489 21948 tablet_service.cc:1505] Processing CreateTablet for tablet 1376a6af91b24e46859bf630618fcec3 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251023 02:04:14.524860 21948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1376a6af91b24e46859bf630618fcec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.537467 22025 tablet_service.cc:1505] Processing CreateTablet for tablet 0325fa135bce480eaaa5faf74daf54f8 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251023 02:04:14.539073 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0325fa135bce480eaaa5faf74daf54f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.542582 22097 tablet_service.cc:1505] Processing CreateTablet for tablet 1376a6af91b24e46859bf630618fcec3 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251023 02:04:14.583755 22097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1376a6af91b24e46859bf630618fcec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.584918 22022 tablet_service.cc:1505] Processing CreateTablet for tablet 8058bbefb0b74ae5949474c893a6947a (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251023 02:04:14.585319 22019 tablet_service.cc:1505] Processing CreateTablet for tablet edc3c9b5058946bf8f0da8fe69900291 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251023 02:04:14.548322 21947 tablet_service.cc:1505] Processing CreateTablet for tablet ffe0e348fc464e36be33082ddcdba159 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251023 02:04:14.551679 21943 tablet_service.cc:1505] Processing CreateTablet for tablet 422ac2af786e439185d0e762e06ef0a3 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251023 02:04:14.551031 21946 tablet_service.cc:1505] Processing CreateTablet for tablet 5e923b66ccf04dd8985bbfe31adbf0b4 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251023 02:04:14.552476 22021 tablet_service.cc:1505] Processing CreateTablet for tablet 5e923b66ccf04dd8985bbfe31adbf0b4 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251023 02:04:14.552940 22152 tablet_bootstrap.cc:492] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:14.592670 22027 tablet_service.cc:1505] Processing CreateTablet for tablet 98f1206f4e204b72af762bd083d766fd (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251023 02:04:14.558082 22151 tablet_bootstrap.cc:492] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:14.564076 21949 tablet_service.cc:1505] Processing CreateTablet for tablet 864751dbbbb748769206d89ccd8cb70e (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251023 02:04:14.566965 22024 tablet_service.cc:1505] Processing CreateTablet for tablet ffe0e348fc464e36be33082ddcdba159 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251023 02:04:14.569203 21950 tablet_service.cc:1505] Processing CreateTablet for tablet edc3c9b5058946bf8f0da8fe69900291 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251023 02:04:14.569316 22094 tablet_service.cc:1505] Processing CreateTablet for tablet 12db617f200e4104b5346432288ad306 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251023 02:04:14.595599 22094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12db617f200e4104b5346432288ad306. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.574368 21952 tablet_service.cc:1505] Processing CreateTablet for tablet 0325fa135bce480eaaa5faf74daf54f8 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251023 02:04:14.576313 22096 tablet_service.cc:1505] Processing CreateTablet for tablet 0325fa135bce480eaaa5faf74daf54f8 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251023 02:04:14.579144 22023 tablet_service.cc:1505] Processing CreateTablet for tablet 422ac2af786e439185d0e762e06ef0a3 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251023 02:04:14.582829 22153 tablet_bootstrap.cc:492] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:14.606096 22021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e923b66ccf04dd8985bbfe31adbf0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.607539 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8058bbefb0b74ae5949474c893a6947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.609287 22152 tablet_bootstrap.cc:654] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:14.610201 22100 tablet_service.cc:1505] Processing CreateTablet for tablet 774c010489aa4775bb6d636926900529 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251023 02:04:14.611338 22027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f1206f4e204b72af762bd083d766fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.612901 22153 tablet_bootstrap.cc:654] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:14.620110 22100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774c010489aa4775bb6d636926900529. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.627664 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe0e348fc464e36be33082ddcdba159. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.581657 22018 tablet_service.cc:1505] Processing CreateTablet for tablet 864751dbbbb748769206d89ccd8cb70e (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251023 02:04:14.631778 22018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 864751dbbbb748769206d89ccd8cb70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.634224 21944 tablet_service.cc:1505] Processing CreateTablet for tablet 8058bbefb0b74ae5949474c893a6947a (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251023 02:04:14.635270 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc3c9b5058946bf8f0da8fe69900291. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.639447 22096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0325fa135bce480eaaa5faf74daf54f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.638549 22020 tablet_service.cc:1505] Processing CreateTablet for tablet 3c29a39800fe46e9b0b85ea73ecc703f (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251023 02:04:14.643537 22020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c29a39800fe46e9b0b85ea73ecc703f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.644454 22151 tablet_bootstrap.cc:654] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:14.647297 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422ac2af786e439185d0e762e06ef0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.648171 22092 tablet_service.cc:1505] Processing CreateTablet for tablet 5e923b66ccf04dd8985bbfe31adbf0b4 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251023 02:04:14.650157 22092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e923b66ccf04dd8985bbfe31adbf0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.657050 22093 tablet_service.cc:1505] Processing CreateTablet for tablet ffe0e348fc464e36be33082ddcdba159 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251023 02:04:14.658314 22093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe0e348fc464e36be33082ddcdba159. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.666384 22021 tablet_service.cc:1505] Processing CreateTablet for tablet c4c983d0648c4b649e08012b7932a15a (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251023 02:04:14.666525 21951 tablet_service.cc:1505] Processing CreateTablet for tablet 98f1206f4e204b72af762bd083d766fd (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251023 02:04:14.667603 22021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c983d0648c4b649e08012b7932a15a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 02:04:14.679119 21908 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251023 02:04:14.689972 22151 tablet_bootstrap.cc:492] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:14.690460 22151 ts_tablet_manager.cc:1403] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.133s user 0.017s sys 0.000s
I20251023 02:04:14.691836 22097 tablet_service.cc:1505] Processing CreateTablet for tablet 864751dbbbb748769206d89ccd8cb70e (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251023 02:04:14.693182 22151 raft_consensus.cc:359] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:14.697180 22151 raft_consensus.cc:385] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:14.697490 22151 raft_consensus.cc:740] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:14.698187 22151 consensus_queue.cc:260] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:14.698601 22099 tablet_service.cc:1505] Processing CreateTablet for tablet edc3c9b5058946bf8f0da8fe69900291 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251023 02:04:14.701205 21945 tablet_service.cc:1505] Processing CreateTablet for tablet 3c29a39800fe46e9b0b85ea73ecc703f (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251023 02:04:14.702428 21945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c29a39800fe46e9b0b85ea73ecc703f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.706918 21948 tablet_service.cc:1505] Processing CreateTablet for tablet c4c983d0648c4b649e08012b7932a15a (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251023 02:04:14.708132 21948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c983d0648c4b649e08012b7932a15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.710619 21944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8058bbefb0b74ae5949474c893a6947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.712303 21952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0325fa135bce480eaaa5faf74daf54f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.713400 22097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 864751dbbbb748769206d89ccd8cb70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.718089 21943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422ac2af786e439185d0e762e06ef0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.721758 21946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e923b66ccf04dd8985bbfe31adbf0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.681598 22098 tablet_service.cc:1505] Processing CreateTablet for tablet 422ac2af786e439185d0e762e06ef0a3 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251023 02:04:14.726496 22098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422ac2af786e439185d0e762e06ef0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.734258 22094 tablet_service.cc:1505] Processing CreateTablet for tablet 8058bbefb0b74ae5949474c893a6947a (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251023 02:04:14.734717 22099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc3c9b5058946bf8f0da8fe69900291. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.735507 22094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8058bbefb0b74ae5949474c893a6947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.736799 21950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc3c9b5058946bf8f0da8fe69900291. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.737499 21951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f1206f4e204b72af762bd083d766fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.744277 21949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 864751dbbbb748769206d89ccd8cb70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.755549 21947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe0e348fc464e36be33082ddcdba159. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.759073 22153 tablet_bootstrap.cc:492] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:14.759569 22153 ts_tablet_manager.cc:1403] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.177s user 0.015s sys 0.025s
I20251023 02:04:14.763095 22152 tablet_bootstrap.cc:492] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:14.763590 22152 ts_tablet_manager.cc:1403] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.211s user 0.007s sys 0.005s
I20251023 02:04:14.768364 22062 heartbeater.cc:499] Master 127.20.81.62:41229 was elected leader, sending a full tablet report...
I20251023 02:04:14.768573 22024 tablet_service.cc:1505] Processing CreateTablet for tablet d80086b174ba4cee80e75f95401994e3 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251023 02:04:14.769903 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d80086b174ba4cee80e75f95401994e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.774441 22022 tablet_service.cc:1505] Processing CreateTablet for tablet c0a3abec901e4c2eba45233d075d9263 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251023 02:04:14.775689 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0a3abec901e4c2eba45233d075d9263. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.775722 22027 tablet_service.cc:1505] Processing CreateTablet for tablet e8a971ac884645548d12e7c83cc039bc (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251023 02:04:14.776844 22093 tablet_service.cc:1505] Processing CreateTablet for tablet 98f1206f4e204b72af762bd083d766fd (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251023 02:04:14.782023 22093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f1206f4e204b72af762bd083d766fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.778458 22101 tablet_service.cc:1505] Processing CreateTablet for tablet c4c983d0648c4b649e08012b7932a15a (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251023 02:04:14.793638 22101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c983d0648c4b649e08012b7932a15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.794155 22153 raft_consensus.cc:359] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:14.777034 22027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8a971ac884645548d12e7c83cc039bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.795590 22092 tablet_service.cc:1505] Processing CreateTablet for tablet 90dcf9b1f8874437a7cd406b2ace7eeb (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251023 02:04:14.794442 22152 raft_consensus.cc:359] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:14.797741 22152 raft_consensus.cc:385] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:14.797778 22153 raft_consensus.cc:385] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:14.798213 22152 raft_consensus.cc:740] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:14.798357 22153 raft_consensus.cc:740] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:14.794911 22021 tablet_service.cc:1505] Processing CreateTablet for tablet 7ba6e9fe4bb44e559397893fa9d899db (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251023 02:04:14.799109 22152 consensus_queue.cc:260] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:14.803524 22153 consensus_queue.cc:260] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:14.811942 22092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90dcf9b1f8874437a7cd406b2ace7eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.816722 21944 tablet_service.cc:1505] Processing CreateTablet for tablet d80086b174ba4cee80e75f95401994e3 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251023 02:04:14.813881 21951 tablet_service.cc:1505] Processing CreateTablet for tablet 90dcf9b1f8874437a7cd406b2ace7eeb (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251023 02:04:14.823096 22098 tablet_service.cc:1505] Processing CreateTablet for tablet d80086b174ba4cee80e75f95401994e3 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251023 02:04:14.823683 22095 tablet_service.cc:1505] Processing CreateTablet for tablet c0a3abec901e4c2eba45233d075d9263 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251023 02:04:14.824400 22098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d80086b174ba4cee80e75f95401994e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.826079 22023 tablet_service.cc:1505] Processing CreateTablet for tablet f932b6202f5b497e8fb358af3c5d3f66 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251023 02:04:14.827448 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f932b6202f5b497e8fb358af3c5d3f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.831881 22021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba6e9fe4bb44e559397893fa9d899db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.831842 22019 tablet_service.cc:1505] Processing CreateTablet for tablet 770eed9a6198463c889a1b9a978bd182 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251023 02:04:14.833376 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770eed9a6198463c889a1b9a978bd182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.793256 22097 tablet_service.cc:1505] Processing CreateTablet for tablet 3c29a39800fe46e9b0b85ea73ecc703f (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251023 02:04:14.846231 22095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0a3abec901e4c2eba45233d075d9263. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.844420 22026 tablet_service.cc:1505] Processing CreateTablet for tablet 90dcf9b1f8874437a7cd406b2ace7eeb (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251023 02:04:14.846935 21950 tablet_service.cc:1505] Processing CreateTablet for tablet e8a971ac884645548d12e7c83cc039bc (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251023 02:04:14.847558 22026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90dcf9b1f8874437a7cd406b2ace7eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.850582 22152 ts_tablet_manager.cc:1434] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.087s user 0.012s sys 0.005s
I20251023 02:04:14.851366 21987 heartbeater.cc:499] Master 127.20.81.62:41229 was elected leader, sending a full tablet report...
I20251023 02:04:14.857438 22151 ts_tablet_manager.cc:1434] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.167s user 0.024s sys 0.020s
I20251023 02:04:14.863722 21945 tablet_service.cc:1505] Processing CreateTablet for tablet c0a3abec901e4c2eba45233d075d9263 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251023 02:04:14.868342 22097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c29a39800fe46e9b0b85ea73ecc703f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.868564 21946 tablet_service.cc:1505] Processing CreateTablet for tablet 770eed9a6198463c889a1b9a978bd182 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251023 02:04:14.873486 22018 tablet_service.cc:1505] Processing CreateTablet for tablet 0dc4e5ba4d0d4809a19f394786d00288 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251023 02:04:14.874714 22018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc4e5ba4d0d4809a19f394786d00288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.876819 22094 tablet_service.cc:1505] Processing CreateTablet for tablet e8a971ac884645548d12e7c83cc039bc (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251023 02:04:14.878008 22094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8a971ac884645548d12e7c83cc039bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.891605 21952 tablet_service.cc:1505] Processing CreateTablet for tablet 0dc4e5ba4d0d4809a19f394786d00288 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251023 02:04:14.898012 22151 tablet_bootstrap.cc:492] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:14.851541 22152 tablet_bootstrap.cc:492] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:14.899675 22025 tablet_service.cc:1505] Processing CreateTablet for tablet 97dae00618e34f58962a1dfbaa38fcdc (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251023 02:04:14.900969 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97dae00618e34f58962a1dfbaa38fcdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.904829 22151 tablet_bootstrap.cc:654] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:14.907882 22100 tablet_service.cc:1505] Processing CreateTablet for tablet 7ba6e9fe4bb44e559397893fa9d899db (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251023 02:04:14.909344 22020 tablet_service.cc:1505] Processing CreateTablet for tablet 4a43ae59c3e347fd86371312be59bb23 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251023 02:04:14.910630 22020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a43ae59c3e347fd86371312be59bb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.911065 21943 tablet_service.cc:1505] Processing CreateTablet for tablet 7ba6e9fe4bb44e559397893fa9d899db (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251023 02:04:14.911559 21947 tablet_service.cc:1505] Processing CreateTablet for tablet f932b6202f5b497e8fb358af3c5d3f66 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251023 02:04:14.912964 22096 tablet_service.cc:1505] Processing CreateTablet for tablet 770eed9a6198463c889a1b9a978bd182 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251023 02:04:14.922617 22019 tablet_service.cc:1505] Processing CreateTablet for tablet 6ead1daa70a847959916a302d8540676 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251023 02:04:14.923830 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ead1daa70a847959916a302d8540676. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.925395 21948 tablet_service.cc:1505] Processing CreateTablet for tablet 97dae00618e34f58962a1dfbaa38fcdc (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251023 02:04:14.926597 21944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d80086b174ba4cee80e75f95401994e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.930932 21950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8a971ac884645548d12e7c83cc039bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.909274 22100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba6e9fe4bb44e559397893fa9d899db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.938762 22136 heartbeater.cc:499] Master 127.20.81.62:41229 was elected leader, sending a full tablet report...
W20251023 02:04:14.939905 21846 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 49700.1us
I20251023 02:04:14.941726 22095 tablet_service.cc:1505] Processing CreateTablet for tablet f932b6202f5b497e8fb358af3c5d3f66 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251023 02:04:14.941463 21949 tablet_service.cc:1505] Processing CreateTablet for tablet 4a43ae59c3e347fd86371312be59bb23 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251023 02:04:14.943413 22098 tablet_service.cc:1505] Processing CreateTablet for tablet 0dc4e5ba4d0d4809a19f394786d00288 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251023 02:04:14.945277 21952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc4e5ba4d0d4809a19f394786d00288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.945969 21945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0a3abec901e4c2eba45233d075d9263. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.948491 21951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90dcf9b1f8874437a7cd406b2ace7eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.949857 21943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba6e9fe4bb44e559397893fa9d899db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.952595 21946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770eed9a6198463c889a1b9a978bd182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.956236 22153 ts_tablet_manager.cc:1434] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.196s user 0.028s sys 0.055s
I20251023 02:04:14.957226 22153 tablet_bootstrap.cc:492] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:14.957854 22096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770eed9a6198463c889a1b9a978bd182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.961874 22098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc4e5ba4d0d4809a19f394786d00288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.965854 22095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f932b6202f5b497e8fb358af3c5d3f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:14.984383 22099 tablet_service.cc:1505] Processing CreateTablet for tablet 97dae00618e34f58962a1dfbaa38fcdc (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251023 02:04:14.985044 22158 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:14.993845 22158 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:14.995173 22153 tablet_bootstrap.cc:654] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:14.995671 22151 tablet_bootstrap.cc:492] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:14.996143 22151 ts_tablet_manager.cc:1403] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.098s user 0.009s sys 0.003s
I20251023 02:04:14.957903 21948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97dae00618e34f58962a1dfbaa38fcdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.014210 22099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97dae00618e34f58962a1dfbaa38fcdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.013875 22094 tablet_service.cc:1505] Processing CreateTablet for tablet 4a43ae59c3e347fd86371312be59bb23 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251023 02:04:15.050930 22152 tablet_bootstrap.cc:654] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.024031 22023 tablet_service.cc:1505] Processing CreateTablet for tablet bd7f335f5df946c481e05b1a2a95f182 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251023 02:04:15.024093 21945 tablet_service.cc:1505] Processing CreateTablet for tablet 6ead1daa70a847959916a302d8540676 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251023 02:04:15.034183 21947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f932b6202f5b497e8fb358af3c5d3f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.075587 22027 tablet_service.cc:1505] Processing CreateTablet for tablet 40b987b2b0ed4294b7ccf057ea33c0a0 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251023 02:04:15.076862 21945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ead1daa70a847959916a302d8540676. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.077338 21952 tablet_service.cc:1505] Processing CreateTablet for tablet 40b987b2b0ed4294b7ccf057ea33c0a0 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251023 02:04:14.996140 22158 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:15.078709 21952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40b987b2b0ed4294b7ccf057ea33c0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.078605 21950 tablet_service.cc:1505] Processing CreateTablet for tablet bd7f335f5df946c481e05b1a2a95f182 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251023 02:04:15.079779 21950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd7f335f5df946c481e05b1a2a95f182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.083297 22094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a43ae59c3e347fd86371312be59bb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.031788 21946 tablet_service.cc:1505] Processing CreateTablet for tablet 1aaf7b9acded449f9017f47455026853 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251023 02:04:15.011464 22018 tablet_service.cc:1505] Processing CreateTablet for tablet 1aaf7b9acded449f9017f47455026853 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251023 02:04:15.027537 22024 tablet_service.cc:1505] Processing CreateTablet for tablet cf578f7b0cdb4188bb43d676b1c64e28 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251023 02:04:15.064110 22101 tablet_service.cc:1505] Processing CreateTablet for tablet 6ead1daa70a847959916a302d8540676 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251023 02:04:15.067201 22151 raft_consensus.cc:359] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.069113 22092 tablet_service.cc:1505] Processing CreateTablet for tablet 1aaf7b9acded449f9017f47455026853 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251023 02:04:15.090544 22101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ead1daa70a847959916a302d8540676. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.073796 22019 tablet_service.cc:1505] Processing CreateTablet for tablet e32bb6c0622f44f8806f77eee0691856 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251023 02:04:15.074290 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd7f335f5df946c481e05b1a2a95f182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.090163 22151 raft_consensus.cc:385] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.092432 22151 raft_consensus.cc:740] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.093216 22151 consensus_queue.cc:260] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.095588 22151 ts_tablet_manager.cc:1434] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.099s user 0.007s sys 0.000s
I20251023 02:04:15.096484 22151 tablet_bootstrap.cc:492] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:15.097862 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32bb6c0622f44f8806f77eee0691856. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.098745 22018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aaf7b9acded449f9017f47455026853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.099051 22092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aaf7b9acded449f9017f47455026853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.103895 22093 tablet_service.cc:1505] Processing CreateTablet for tablet 40b987b2b0ed4294b7ccf057ea33c0a0 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251023 02:04:15.118463 22093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40b987b2b0ed4294b7ccf057ea33c0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.093152 21949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a43ae59c3e347fd86371312be59bb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.123786 21951 tablet_service.cc:1505] Processing CreateTablet for tablet e32bb6c0622f44f8806f77eee0691856 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251023 02:04:15.131578 21950 tablet_service.cc:1505] Processing CreateTablet for tablet cf578f7b0cdb4188bb43d676b1c64e28 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251023 02:04:15.134693 22098 tablet_service.cc:1505] Processing CreateTablet for tablet e32bb6c0622f44f8806f77eee0691856 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251023 02:04:15.147360 22098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32bb6c0622f44f8806f77eee0691856. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.148643 21944 tablet_service.cc:1505] Processing CreateTablet for tablet 5470b0fad25247c19157cd390de47f7f (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251023 02:04:15.139436 22099 tablet_service.cc:1505] Processing CreateTablet for tablet 5470b0fad25247c19157cd390de47f7f (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251023 02:04:15.144622 22151 tablet_bootstrap.cc:654] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.153133 22099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5470b0fad25247c19157cd390de47f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.162703 22100 tablet_service.cc:1505] Processing CreateTablet for tablet 972d07041fbd4097854ef23c73f09c75 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251023 02:04:15.157058 22153 tablet_bootstrap.cc:492] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:15.204562 22153 ts_tablet_manager.cc:1403] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.248s user 0.013s sys 0.015s
I20251023 02:04:15.205508 22094 tablet_service.cc:1505] Processing CreateTablet for tablet 00301d3b4a804bf19bad66c51fe19faa (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251023 02:04:15.206820 22094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00301d3b4a804bf19bad66c51fe19faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.207383 22153 raft_consensus.cc:359] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.208082 22153 raft_consensus.cc:385] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.208431 22153 raft_consensus.cc:740] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.157653 22025 tablet_service.cc:1505] Processing CreateTablet for tablet 5470b0fad25247c19157cd390de47f7f (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251023 02:04:15.132859 22097 tablet_service.cc:1505] Processing CreateTablet for tablet cf578f7b0cdb4188bb43d676b1c64e28 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251023 02:04:15.210131 21945 tablet_service.cc:1505] Processing CreateTablet for tablet 74071b399ecc41808f0ac84dc836af4c (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251023 02:04:15.210896 22097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf578f7b0cdb4188bb43d676b1c64e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.212739 21947 tablet_service.cc:1505] Processing CreateTablet for tablet 93b201674548485eaa29b78067f74a66 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251023 02:04:15.153424 22027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40b987b2b0ed4294b7ccf057ea33c0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.155016 21946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aaf7b9acded449f9017f47455026853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.216006 21947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93b201674548485eaa29b78067f74a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.159106 22152 tablet_bootstrap.cc:492] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:15.217867 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5470b0fad25247c19157cd390de47f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.217932 22152 ts_tablet_manager.cc:1403] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.367s user 0.013s sys 0.012s
I20251023 02:04:15.220765 22152 raft_consensus.cc:359] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.167506 22020 tablet_service.cc:1505] Processing CreateTablet for tablet 972d07041fbd4097854ef23c73f09c75 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251023 02:04:15.222203 21945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74071b399ecc41808f0ac84dc836af4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.222332 22020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 972d07041fbd4097854ef23c73f09c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.167732 22026 tablet_service.cc:1505] Processing CreateTablet for tablet 74071b399ecc41808f0ac84dc836af4c (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251023 02:04:15.168939 22021 tablet_service.cc:1505] Processing CreateTablet for tablet 6dddaeefc3d449a0ba9b3e6695e37507 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251023 02:04:15.169014 22023 tablet_service.cc:1505] Processing CreateTablet for tablet 93b201674548485eaa29b78067f74a66 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251023 02:04:15.170936 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:15.174090 22098 tablet_service.cc:1505] Processing CreateTablet for tablet 74071b399ecc41808f0ac84dc836af4c (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251023 02:04:15.226493 22026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74071b399ecc41808f0ac84dc836af4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.226718 22037 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:15.227236 22098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74071b399ecc41808f0ac84dc836af4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.227754 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93b201674548485eaa29b78067f74a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.130805 22095 tablet_service.cc:1505] Processing CreateTablet for tablet bd7f335f5df946c481e05b1a2a95f182 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251023 02:04:15.229907 22095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd7f335f5df946c481e05b1a2a95f182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.230098 22021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dddaeefc3d449a0ba9b3e6695e37507. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.230245 21928 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:15.180547 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:15.231230 22158 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:15.231448 22111 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:15.231940 21944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5470b0fad25247c19157cd390de47f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.184173 22096 tablet_service.cc:1505] Processing CreateTablet for tablet 6dddaeefc3d449a0ba9b3e6695e37507 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251023 02:04:15.190348 22157 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:15.209141 22153 consensus_queue.cc:260] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.232975 22096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dddaeefc3d449a0ba9b3e6695e37507. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.232648 22157 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:15.188156 22099 tablet_service.cc:1505] Processing CreateTablet for tablet 93b201674548485eaa29b78067f74a66 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251023 02:04:15.235069 22099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93b201674548485eaa29b78067f74a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.191296 21948 tablet_service.cc:1505] Processing CreateTablet for tablet 972d07041fbd4097854ef23c73f09c75 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251023 02:04:15.200363 22022 tablet_service.cc:1505] Processing CreateTablet for tablet 00301d3b4a804bf19bad66c51fe19faa (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251023 02:04:15.240948 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00301d3b4a804bf19bad66c51fe19faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.231598 22158 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:15.242837 22158 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:15.198514 21943 tablet_service.cc:1505] Processing CreateTablet for tablet 6dddaeefc3d449a0ba9b3e6695e37507 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251023 02:04:15.244643 21943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dddaeefc3d449a0ba9b3e6695e37507. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.246495 21950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf578f7b0cdb4188bb43d676b1c64e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.255739 22153 ts_tablet_manager.cc:1434] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.051s user 0.007s sys 0.000s
I20251023 02:04:15.221469 22152 raft_consensus.cc:385] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.260393 22100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 972d07041fbd4097854ef23c73f09c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.262614 22152 raft_consensus.cc:740] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.263650 21948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 972d07041fbd4097854ef23c73f09c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.265236 22157 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:15.266120 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf578f7b0cdb4188bb43d676b1c64e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.277268 22094 tablet_service.cc:1505] Processing CreateTablet for tablet d2b6e2b19ddc453290a76212731a2f28 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251023 02:04:15.278591 22094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b6e2b19ddc453290a76212731a2f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.269865 22152 consensus_queue.cc:260] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.298293 21951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32bb6c0622f44f8806f77eee0691856. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.297760 22153 tablet_bootstrap.cc:492] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:15.326790 22158 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.328907 22158 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:15.329568 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:15.343266 22037 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:15.329950 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:15.338456 22101 tablet_service.cc:1505] Processing CreateTablet for tablet fb1e5f57f70948bcb887f55cb3702b0e (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251023 02:04:15.345930 22111 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:15.346607 22101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb1e5f57f70948bcb887f55cb3702b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.330982 22092 tablet_service.cc:1505] Processing CreateTablet for tablet 496e171acd9c4d6d9dab2ab594c06e57 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251023 02:04:15.341857 22151 tablet_bootstrap.cc:492] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:15.349946 22151 ts_tablet_manager.cc:1403] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.254s user 0.013s sys 0.004s
I20251023 02:04:15.350306 22092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 496e171acd9c4d6d9dab2ab594c06e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.358260 22151 raft_consensus.cc:359] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.358964 22151 raft_consensus.cc:385] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.359129 22111 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:15.359280 22151 raft_consensus.cc:740] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.360574 21925 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:15.360415 22151 consensus_queue.cc:260] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.363134 22151 ts_tablet_manager.cc:1434] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20251023 02:04:15.364095 22151 tablet_bootstrap.cc:492] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:15.372740 22019 tablet_service.cc:1505] Processing CreateTablet for tablet d2b6e2b19ddc453290a76212731a2f28 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251023 02:04:15.375381 22151 tablet_bootstrap.cc:654] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.376050 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:15.378455 22023 tablet_service.cc:1505] Processing CreateTablet for tablet fb1e5f57f70948bcb887f55cb3702b0e (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251023 02:04:15.399097 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb1e5f57f70948bcb887f55cb3702b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.401087 22020 tablet_service.cc:1505] Processing CreateTablet for tablet 816ef481cee84b259bc409b45a81baf1 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251023 02:04:15.402328 22020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 816ef481cee84b259bc409b45a81baf1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 02:04:15.406345 22000 leader_election.cc:343] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:15.410562 22097 tablet_service.cc:1505] Processing CreateTablet for tablet de76561746ce42549f51431c3f40bed7 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251023 02:04:15.411509 22021 tablet_service.cc:1505] Processing CreateTablet for tablet de76561746ce42549f51431c3f40bed7 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251023 02:04:15.412675 22021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de76561746ce42549f51431c3f40bed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.411878 22097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de76561746ce42549f51431c3f40bed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.380056 22037 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:15.389968 22152 ts_tablet_manager.cc:1434] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.172s user 0.003s sys 0.009s
I20251023 02:04:15.415349 22153 tablet_bootstrap.cc:654] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.414870 22096 tablet_service.cc:1505] Processing CreateTablet for tablet 816ef481cee84b259bc409b45a81baf1 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251023 02:04:15.417702 22152 tablet_bootstrap.cc:492] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:15.418571 21947 tablet_service.cc:1505] Processing CreateTablet for tablet fb1e5f57f70948bcb887f55cb3702b0e (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251023 02:04:15.419835 21947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb1e5f57f70948bcb887f55cb3702b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.419593 21946 tablet_service.cc:1505] Processing CreateTablet for tablet 496e171acd9c4d6d9dab2ab594c06e57 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251023 02:04:15.420840 21946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 496e171acd9c4d6d9dab2ab594c06e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.395583 22027 tablet_service.cc:1505] Processing CreateTablet for tablet 496e171acd9c4d6d9dab2ab594c06e57 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251023 02:04:15.422618 22027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 496e171acd9c4d6d9dab2ab594c06e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.341212 21952 tablet_service.cc:1505] Processing CreateTablet for tablet 00301d3b4a804bf19bad66c51fe19faa (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251023 02:04:15.426041 21952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00301d3b4a804bf19bad66c51fe19faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.397316 21943 tablet_service.cc:1505] Processing CreateTablet for tablet d2b6e2b19ddc453290a76212731a2f28 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251023 02:04:15.430286 21943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b6e2b19ddc453290a76212731a2f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.432613 21949 tablet_service.cc:1505] Processing CreateTablet for tablet 816ef481cee84b259bc409b45a81baf1 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251023 02:04:15.433830 21949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 816ef481cee84b259bc409b45a81baf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.396828 22158 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:15.438095 22158 raft_consensus.cc:697] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:15.439566 22158 consensus_queue.cc:237] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.441865 22096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 816ef481cee84b259bc409b45a81baf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.425020 22152 tablet_bootstrap.cc:654] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.446960 21944 tablet_service.cc:1505] Processing CreateTablet for tablet de76561746ce42549f51431c3f40bed7 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251023 02:04:15.452222 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:15.457811 22002 leader_election.cc:343] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:15.458236 22002 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:15.458849 21948 tablet_service.cc:1505] Processing CreateTablet for tablet 7600022898784926af7a5289ce03bc3a (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251023 02:04:15.459453 22157 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:15.466825 21948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7600022898784926af7a5289ce03bc3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.482749 22099 tablet_service.cc:1505] Processing CreateTablet for tablet 7600022898784926af7a5289ce03bc3a (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251023 02:04:15.483953 22024 tablet_service.cc:1505] Processing CreateTablet for tablet 6869bc9cdea241aaa6d86d46a1b5bcfb (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251023 02:04:15.484855 22099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7600022898784926af7a5289ce03bc3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.486055 22018 tablet_service.cc:1505] Processing CreateTablet for tablet 7600022898784926af7a5289ce03bc3a (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251023 02:04:15.487207 21858 catalog_manager.cc:5649] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:15.480042 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b6e2b19ddc453290a76212731a2f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.495373 22098 tablet_service.cc:1505] Processing CreateTablet for tablet 6869bc9cdea241aaa6d86d46a1b5bcfb (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251023 02:04:15.496611 22098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6869bc9cdea241aaa6d86d46a1b5bcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.505710 21944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de76561746ce42549f51431c3f40bed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.506296 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6869bc9cdea241aaa6d86d46a1b5bcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.511140 21943 tablet_service.cc:1505] Processing CreateTablet for tablet fe5d794166cc4fbb87c6b918fa26b592 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251023 02:04:15.512416 21943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe5d794166cc4fbb87c6b918fa26b592. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.512593 22023 tablet_service.cc:1505] Processing CreateTablet for tablet fe5d794166cc4fbb87c6b918fa26b592 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251023 02:04:15.513882 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe5d794166cc4fbb87c6b918fa26b592. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.514587 22094 tablet_service.cc:1505] Processing CreateTablet for tablet fe5d794166cc4fbb87c6b918fa26b592 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251023 02:04:15.515789 22094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe5d794166cc4fbb87c6b918fa26b592. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.521844 22018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7600022898784926af7a5289ce03bc3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.528052 22159 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:15.528537 22159 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.544004 21950 tablet_service.cc:1505] Processing CreateTablet for tablet 6869bc9cdea241aaa6d86d46a1b5bcfb (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251023 02:04:15.544992 22159 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:15.545336 21950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6869bc9cdea241aaa6d86d46a1b5bcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.562405 22151 tablet_bootstrap.cc:492] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:15.562330 21945 tablet_service.cc:1505] Processing CreateTablet for tablet d2ed2c541deb4c8eb9997f54a56f68d7 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251023 02:04:15.562947 22151 ts_tablet_manager.cc:1403] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.199s user 0.008s sys 0.028s
I20251023 02:04:15.563742 21945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2ed2c541deb4c8eb9997f54a56f68d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.566893 22151 raft_consensus.cc:359] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.567602 22151 raft_consensus.cc:385] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.567915 22151 raft_consensus.cc:740] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.568686 22151 consensus_queue.cc:260] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.570653 22023 tablet_service.cc:1505] Processing CreateTablet for tablet d2ed2c541deb4c8eb9997f54a56f68d7 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251023 02:04:15.572084 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2ed2c541deb4c8eb9997f54a56f68d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.575189 22151 ts_tablet_manager.cc:1434] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.012s user 0.004s sys 0.003s
I20251023 02:04:15.576525 21951 tablet_service.cc:1505] Processing CreateTablet for tablet c954cadc886d47b39ed2cf51f9ac59f7 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251023 02:04:15.577785 21951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c954cadc886d47b39ed2cf51f9ac59f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.579476 22022 tablet_service.cc:1505] Processing CreateTablet for tablet c954cadc886d47b39ed2cf51f9ac59f7 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251023 02:04:15.580685 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c954cadc886d47b39ed2cf51f9ac59f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.581955 22158 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:15.582424 22158 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.583817 22021 tablet_service.cc:1505] Processing CreateTablet for tablet f62998a4a7ab4a4a878db25922f83a69 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251023 02:04:15.584319 22158 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:15.584954 22021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f62998a4a7ab4a4a878db25922f83a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.590517 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:15.591820 21925 leader_election.cc:343] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:15.596604 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:15.606340 21948 tablet_service.cc:1505] Processing CreateTablet for tablet f62998a4a7ab4a4a878db25922f83a69 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251023 02:04:15.607589 21948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f62998a4a7ab4a4a878db25922f83a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.609006 21952 tablet_service.cc:1505] Processing CreateTablet for tablet 2da2ca0f34944635a558b8678ecc206f (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251023 02:04:15.615142 22020 tablet_service.cc:1505] Processing CreateTablet for tablet 2da2ca0f34944635a558b8678ecc206f (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251023 02:04:15.626135 22151 tablet_bootstrap.cc:492] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
W20251023 02:04:15.621541 21928 leader_election.cc:343] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:15.637773 21928 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:15.626346 21952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da2ca0f34944635a558b8678ecc206f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.628866 22152 tablet_bootstrap.cc:492] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:15.631243 22026 tablet_service.cc:1505] Processing CreateTablet for tablet 66c8e1c6015d47879ab7927548ad3b02 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251023 02:04:15.633831 21944 tablet_service.cc:1505] Processing CreateTablet for tablet 66c8e1c6015d47879ab7927548ad3b02 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251023 02:04:15.638618 22158 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:15.652709 22025 tablet_service.cc:1505] Processing CreateTablet for tablet f745a2e5ae9f4a829fa01ad2bce79767 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251023 02:04:15.648025 22020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da2ca0f34944635a558b8678ecc206f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.654445 21944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c8e1c6015d47879ab7927548ad3b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.654762 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f745a2e5ae9f4a829fa01ad2bce79767. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.655930 21950 tablet_service.cc:1505] Processing CreateTablet for tablet f745a2e5ae9f4a829fa01ad2bce79767 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251023 02:04:15.648383 22152 ts_tablet_manager.cc:1403] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.231s user 0.013s sys 0.026s
I20251023 02:04:15.660672 22026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c8e1c6015d47879ab7927548ad3b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.668553 22092 tablet_service.cc:1505] Processing CreateTablet for tablet f62998a4a7ab4a4a878db25922f83a69 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251023 02:04:15.669833 22092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f62998a4a7ab4a4a878db25922f83a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.674551 22098 tablet_service.cc:1505] Processing CreateTablet for tablet c954cadc886d47b39ed2cf51f9ac59f7 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251023 02:04:15.677992 22098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c954cadc886d47b39ed2cf51f9ac59f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.678262 22097 tablet_service.cc:1505] Processing CreateTablet for tablet 2da2ca0f34944635a558b8678ecc206f (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251023 02:04:15.679193 22097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da2ca0f34944635a558b8678ecc206f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 02:04:15.681376 21908 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251023 02:04:15.682325 22152 raft_consensus.cc:359] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.682818 22094 tablet_service.cc:1505] Processing CreateTablet for tablet 66c8e1c6015d47879ab7927548ad3b02 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251023 02:04:15.683388 22152 raft_consensus.cc:385] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.683789 22152 raft_consensus.cc:740] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.684584 22152 consensus_queue.cc:260] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.685535 22100 tablet_service.cc:1505] Processing CreateTablet for tablet f745a2e5ae9f4a829fa01ad2bce79767 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251023 02:04:15.683951 22094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c8e1c6015d47879ab7927548ad3b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.702492 22096 tablet_service.cc:1505] Processing CreateTablet for tablet d2ed2c541deb4c8eb9997f54a56f68d7 (DEFAULT_TABLE table=test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251023 02:04:15.703032 22153 tablet_bootstrap.cc:492] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:15.709888 21950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f745a2e5ae9f4a829fa01ad2bce79767. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.712716 22100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f745a2e5ae9f4a829fa01ad2bce79767. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:15.728817 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:15.729899 22096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2ed2c541deb4c8eb9997f54a56f68d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 02:04:15.730221 22077 leader_election.cc:343] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:15.730979 22153 ts_tablet_manager.cc:1403] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.457s user 0.019s sys 0.035s
I20251023 02:04:15.732681 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:15.734160 22076 leader_election.cc:343] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:15.734635 22076 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:15.735440 22159 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:15.736559 22152 ts_tablet_manager.cc:1434] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.078s user 0.000s sys 0.006s
I20251023 02:04:15.737668 22152 tablet_bootstrap.cc:492] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:15.739112 22153 raft_consensus.cc:359] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.739933 22153 raft_consensus.cc:385] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.740236 22153 raft_consensus.cc:740] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.740965 22153 consensus_queue.cc:260] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.745491 22152 tablet_bootstrap.cc:654] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.751073 22153 ts_tablet_manager.cc:1434] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20251023 02:04:15.752241 22153 tablet_bootstrap.cc:492] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:15.755270 22151 tablet_bootstrap.cc:654] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.764430 22153 tablet_bootstrap.cc:654] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.766860 22157 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:15.767499 22157 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.770386 22157 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:15.773156 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:15.773072 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:15.774655 22111 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:15.774992 22152 tablet_bootstrap.cc:492] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:15.775290 22151 tablet_bootstrap.cc:492] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:15.775590 22152 ts_tablet_manager.cc:1403] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.011s
I20251023 02:04:15.775744 22151 ts_tablet_manager.cc:1403] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.150s user 0.009s sys 0.004s
I20251023 02:04:15.775911 22000 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:15.776878 22157 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:15.777240 22157 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:15.777556 22157 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:15.781601 22153 tablet_bootstrap.cc:492] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:15.782053 22153 ts_tablet_manager.cc:1403] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.007s
I20251023 02:04:15.778614 22151 raft_consensus.cc:359] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.783360 22151 raft_consensus.cc:385] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.783694 22151 raft_consensus.cc:740] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
W20251023 02:04:15.780423 22002 leader_election.cc:343] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:15.784641 22153 raft_consensus.cc:359] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.784498 22151 consensus_queue.cc:260] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.779235 22152 raft_consensus.cc:359] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.785184 22153 raft_consensus.cc:385] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.785454 22152 raft_consensus.cc:385] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.785532 22153 raft_consensus.cc:740] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.785936 22152 raft_consensus.cc:740] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.786378 22153 consensus_queue.cc:260] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.786857 22152 consensus_queue.cc:260] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.788686 22153 ts_tablet_manager.cc:1434] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251023 02:04:15.789950 22153 tablet_bootstrap.cc:492] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:15.795267 22152 ts_tablet_manager.cc:1434] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20251023 02:04:15.795986 22153 tablet_bootstrap.cc:654] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.796579 22152 tablet_bootstrap.cc:492] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:15.805289 22152 tablet_bootstrap.cc:654] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.813105 22151 ts_tablet_manager.cc:1434] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.037s user 0.028s sys 0.005s
I20251023 02:04:15.819543 22157 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.823185 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:15.824486 22002 leader_election.cc:343] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:15.825620 22158 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:15.826385 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:15.826913 22111 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:15.828429 22151 tablet_bootstrap.cc:492] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:15.835752 22111 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:15.837311 22151 tablet_bootstrap.cc:654] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.837757 22000 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:15.839138 22185 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:15.836400 22157 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:15.855229 22158 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:15.855695 22158 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.856678 22152 tablet_bootstrap.cc:492] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:15.857193 22152 ts_tablet_manager.cc:1403] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.061s user 0.023s sys 0.029s
I20251023 02:04:15.857674 22158 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:15.859184 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:15.860095 22036 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:15.861054 22185 raft_consensus.cc:697] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:15.861627 22152 raft_consensus.cc:359] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:15.862120 21928 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:15.862421 22152 raft_consensus.cc:385] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.862857 22152 raft_consensus.cc:740] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.862243 22185 consensus_queue.cc:237] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.854903 22153 tablet_bootstrap.cc:492] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:15.863899 22158 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:15.864041 22153 ts_tablet_manager.cc:1403] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.074s user 0.020s sys 0.022s
I20251023 02:04:15.864398 22158 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:15.864892 22158 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:15.865955 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:15.867630 21925 leader_election.cc:343] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:15.869346 22153 raft_consensus.cc:359] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.865989 22152 consensus_queue.cc:260] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:15.874512 22158 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.880788 22151 tablet_bootstrap.cc:492] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:15.870239 22153 raft_consensus.cc:385] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.881824 22151 ts_tablet_manager.cc:1403] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.054s user 0.023s sys 0.018s
I20251023 02:04:15.883162 22158 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:15.881789 22153 raft_consensus.cc:740] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.883872 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:15.884449 22037 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:15.886994 22151 raft_consensus.cc:359] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.890565 22151 raft_consensus.cc:385] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:15.890828 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:15.890955 22151 raft_consensus.cc:740] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:15.884382 22153 consensus_queue.cc:260] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.889621 22152 ts_tablet_manager.cc:1434] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.032s user 0.006s sys 0.009s
I20251023 02:04:15.891964 22151 consensus_queue.cc:260] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:15.892632 21925 leader_election.cc:343] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:15.894464 22151 ts_tablet_manager.cc:1434] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20251023 02:04:15.895421 22151 tablet_bootstrap.cc:492] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:15.895462 22037 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:15.892778 22152 tablet_bootstrap.cc:492] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:15.895690 21860 catalog_manager.cc:5649] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:15.897781 22153 ts_tablet_manager.cc:1434] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20251023 02:04:15.898734 22153 tablet_bootstrap.cc:492] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:15.901528 21928 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:15.902325 22151 tablet_bootstrap.cc:654] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.902325 22152 tablet_bootstrap.cc:654] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.902776 22192 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:15.903280 22192 raft_consensus.cc:697] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:15.904064 22192 consensus_queue.cc:237] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:15.907532 22153 tablet_bootstrap.cc:654] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:15.902487 22158 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:15.991858 21857 catalog_manager.cc:5649] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:15.994056 22158 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:15.994716 22158 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:15.998085 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:15.999462 21925 leader_election.cc:343] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.026214 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.026859 22036 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:16.027715 22158 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.028002 21928 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:16.028991 22185 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.029210 22192 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:16.029557 22192 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:16.029441 22185 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.029922 22192 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.031399 22185 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.033438 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.033946 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:16.034225 21962 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
W20251023 02:04:16.035476 22000 leader_election.cc:343] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.036218 22002 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:16.037684 22185 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:16.038162 22185 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:16.038581 22185 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:16.042057 22063 tablet.cc:2378] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 02:04:16.065044 22189 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.065546 22189 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.067539 22189 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.069914 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.070534 21962 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:16.071453 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:16.071631 22002 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:16.072399 22189 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251023 02:04:16.072597 22000 leader_election.cc:343] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.072765 22189 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:16.073124 22189 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.052991 22185 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.092643 22152 tablet_bootstrap.cc:492] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.092967 22153 tablet_bootstrap.cc:492] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:16.093194 22152 ts_tablet_manager.cc:1403] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.201s user 0.064s sys 0.127s
I20251023 02:04:16.093433 22153 ts_tablet_manager.cc:1403] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.195s user 0.051s sys 0.119s
I20251023 02:04:16.095677 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:16.096251 21962 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.096025 22152 raft_consensus.cc:359] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.096728 22152 raft_consensus.cc:385] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.096290 22153 raft_consensus.cc:359] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.097043 22152 raft_consensus.cc:740] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.097127 22153 raft_consensus.cc:385] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.097327 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:16.097496 22153 raft_consensus.cc:740] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.097903 22152 consensus_queue.cc:260] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.098198 22153 consensus_queue.cc:260] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:16.099125 22000 leader_election.cc:343] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.098613 22159 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.099948 22159 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.102936 22152 ts_tablet_manager.cc:1434] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20251023 02:04:16.103896 22152 tablet_bootstrap.cc:492] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.104332 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.105525 22159 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.106033 21962 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:16.108700 22151 tablet_bootstrap.cc:492] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:16.109279 22151 ts_tablet_manager.cc:1403] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.214s user 0.057s sys 0.129s
I20251023 02:04:16.110042 22152 tablet_bootstrap.cc:654] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.109957 22002 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:16.111306 22076 leader_election.cc:343] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.111512 22157 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:16.114965 22157 raft_consensus.cc:697] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:16.115967 22157 consensus_queue.cc:237] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.120111 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:16.121845 22077 leader_election.cc:343] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.122473 22077 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:16.121706 22151 raft_consensus.cc:359] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.124868 22151 raft_consensus.cc:385] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.129104 22151 raft_consensus.cc:740] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.130100 22151 consensus_queue.cc:260] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.133901 22151 ts_tablet_manager.cc:1434] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20251023 02:04:16.135176 22151 tablet_bootstrap.cc:492] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:16.144935 22151 tablet_bootstrap.cc:654] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.151268 22159 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.140976 21859 catalog_manager.cc:5649] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:16.158099 22185 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.069329 22192 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:16.161424 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:04:16.168417 21925 leader_election.cc:343] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.177011 22189 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.180905 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:16.181154 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:16.181749 21962 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:16.183022 22000 leader_election.cc:343] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.185875 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:16.186677 22036 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.191938 21962 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:16.193595 22002 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:16.194828 22185 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:16.196419 22185 raft_consensus.cc:697] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:16.196934 22036 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:16.197678 22192 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.198740 22189 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.197809 22185 consensus_queue.cc:237] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.200630 21928 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:16.202147 22192 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:16.202773 22192 raft_consensus.cc:697] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:16.203851 22192 consensus_queue.cc:237] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:16.209255 22152 tablet_bootstrap.cc:492] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.210112 22152 ts_tablet_manager.cc:1403] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.106s user 0.033s sys 0.061s
I20251023 02:04:16.214958 22159 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.215539 22159 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.215972 22152 raft_consensus.cc:359] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.216674 22152 raft_consensus.cc:385] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.216969 22152 raft_consensus.cc:740] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.217674 22152 consensus_queue.cc:260] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.221457 22152 ts_tablet_manager.cc:1434] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251023 02:04:16.222697 22152 tablet_bootstrap.cc:492] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.223124 22185 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.223604 22185 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.225443 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.226270 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:16.227669 22002 leader_election.cc:343] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.227934 22159 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.226301 21962 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:16.228929 22152 tablet_bootstrap.cc:654] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.229945 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:16.220575 21859 catalog_manager.cc:5649] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
W20251023 02:04:16.231384 22000 leader_election.cc:343] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.232012 22000 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:16.229839 22076 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:16.233786 22159 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:16.234227 22159 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:16.234670 22159 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.234354 22189 raft_consensus.cc:2749] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.234728 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.242502 22159 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:16.243060 22077 leader_election.cc:343] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.239862 22152 tablet_bootstrap.cc:492] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.244562 22152 ts_tablet_manager.cc:1403] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.003s
I20251023 02:04:16.245153 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:16.245880 21962 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.246207 22153 ts_tablet_manager.cc:1434] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.152s user 0.042s sys 0.104s
I20251023 02:04:16.246026 22202 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.247012 22152 raft_consensus.cc:359] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.247280 22153 tablet_bootstrap.cc:492] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:16.247247 22202 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.247678 22152 raft_consensus.cc:385] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.249950 22152 raft_consensus.cc:740] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.250543 22152 consensus_queue.cc:260] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.252830 22152 ts_tablet_manager.cc:1434] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20251023 02:04:16.253410 21962 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:16.253741 22152 tablet_bootstrap.cc:492] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.253890 22153 tablet_bootstrap.cc:654] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.254528 22076 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:16.257097 22185 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.257282 22210 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:16.259811 22151 tablet_bootstrap.cc:492] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:16.259865 22152 tablet_bootstrap.cc:654] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.260633 22151 ts_tablet_manager.cc:1403] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.126s user 0.022s sys 0.032s
I20251023 02:04:16.261677 22210 raft_consensus.cc:697] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:16.261822 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:16.263629 22077 leader_election.cc:343] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.262874 22159 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.265522 22210 consensus_queue.cc:237] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.271720 22151 raft_consensus.cc:359] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.272187 22151 raft_consensus.cc:385] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.272433 22151 raft_consensus.cc:740] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.273245 22151 consensus_queue.cc:260] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.278529 22202 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.279572 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.276005 21859 catalog_manager.cc:5649] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:16.281409 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.283275 22152 tablet_bootstrap.cc:492] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.283658 22152 ts_tablet_manager.cc:1403] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.006s
W20251023 02:04:16.285408 22077 leader_election.cc:343] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.285617 22152 raft_consensus.cc:359] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.286279 22152 raft_consensus.cc:385] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.286547 22152 raft_consensus.cc:740] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
W20251023 02:04:16.286783 22076 leader_election.cc:343] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.287344 22076 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:16.287139 22152 consensus_queue.cc:260] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.288431 22202 raft_consensus.cc:2749] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.290516 22151 ts_tablet_manager.cc:1434] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20251023 02:04:16.290882 22152 ts_tablet_manager.cc:1434] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251023 02:04:16.291429 22151 tablet_bootstrap.cc:492] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:16.291769 22152 tablet_bootstrap.cc:492] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.291851 22153 tablet_bootstrap.cc:492] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:16.292415 22153 ts_tablet_manager.cc:1403] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.045s user 0.020s sys 0.011s
I20251023 02:04:16.295504 22153 raft_consensus.cc:359] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.296268 22153 raft_consensus.cc:385] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.296605 22153 raft_consensus.cc:740] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.297561 22151 tablet_bootstrap.cc:654] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.297345 22153 consensus_queue.cc:260] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.299963 22153 ts_tablet_manager.cc:1434] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251023 02:04:16.300629 22152 tablet_bootstrap.cc:654] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.301028 22153 tablet_bootstrap.cc:492] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:16.308982 22151 tablet_bootstrap.cc:492] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:16.309494 22151 ts_tablet_manager.cc:1403] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.002s
I20251023 02:04:16.309648 22153 tablet_bootstrap.cc:654] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.311975 22151 raft_consensus.cc:359] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.312647 22151 raft_consensus.cc:385] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.312922 22151 raft_consensus.cc:740] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.313742 22151 consensus_queue.cc:260] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.316045 22151 ts_tablet_manager.cc:1434] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251023 02:04:16.316900 22151 tablet_bootstrap.cc:492] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:16.322356 22151 tablet_bootstrap.cc:654] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.327710 22153 tablet_bootstrap.cc:492] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:16.328383 22153 ts_tablet_manager.cc:1403] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.012s
I20251023 02:04:16.331313 22153 raft_consensus.cc:359] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.331962 22153 raft_consensus.cc:385] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.332237 22153 raft_consensus.cc:740] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.332757 22152 tablet_bootstrap.cc:492] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.333205 22152 ts_tablet_manager.cc:1403] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.020s
I20251023 02:04:16.332868 22153 consensus_queue.cc:260] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.334434 22151 tablet_bootstrap.cc:492] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:16.334858 22151 ts_tablet_manager.cc:1403] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.000s
I20251023 02:04:16.334906 22153 ts_tablet_manager.cc:1434] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251023 02:04:16.335824 22153 tablet_bootstrap.cc:492] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:16.335553 22152 raft_consensus.cc:359] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.336203 22152 raft_consensus.cc:385] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.335470 21859 catalog_manager.cc:5649] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:16.336447 22152 raft_consensus.cc:740] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.337141 22152 consensus_queue.cc:260] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.338817 22158 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.339335 22158 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.339699 22185 consensus_queue.cc:1048] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:16.341238 22158 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.342092 22153 tablet_bootstrap.cc:654] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.346253 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:16.347666 22151 raft_consensus.cc:359] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.348258 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:16.348981 21925 leader_election.cc:343] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
W20251023 02:04:16.349506 21928 leader_election.cc:343] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.349965 21928 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:16.352382 22153 tablet_bootstrap.cc:492] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:16.348264 22151 raft_consensus.cc:385] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.353009 22153 ts_tablet_manager.cc:1403] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20251023 02:04:16.353760 22151 raft_consensus.cc:740] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.354127 22192 raft_consensus.cc:2749] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.355002 22151 consensus_queue.cc:260] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.356145 22152 ts_tablet_manager.cc:1434] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20251023 02:04:16.355947 22153 raft_consensus.cc:359] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.356568 22153 raft_consensus.cc:385] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.356851 22153 raft_consensus.cc:740] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.357100 22111 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.357446 22153 consensus_queue.cc:260] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.359586 22153 ts_tablet_manager.cc:1434] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251023 02:04:16.360474 22153 tablet_bootstrap.cc:492] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:16.362737 22151 ts_tablet_manager.cc:1434] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
I20251023 02:04:16.363611 22151 tablet_bootstrap.cc:492] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:16.364385 22152 tablet_bootstrap.cc:492] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.364487 22185 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.364967 22185 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.365710 22153 tablet_bootstrap.cc:654] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.367533 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.368149 21962 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:16.369354 22002 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:16.370218 22189 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:16.370572 22189 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:16.370887 22189 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.371812 22152 tablet_bootstrap.cc:654] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.375433 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:16.376319 22109 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:16.376911 22185 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.378546 22202 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.379043 22202 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.380445 22192 consensus_queue.cc:1048] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:16.384441 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.385076 22036 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:16.386245 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.388088 22077 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:16.389312 22202 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.393815 22192 consensus_queue.cc:1048] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:16.398217 22152 tablet_bootstrap.cc:492] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.398813 22152 ts_tablet_manager.cc:1403] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.002s
I20251023 02:04:16.400336 22189 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:16.401674 22002 consensus_peers.cc:597] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 -> Peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Couldn't send request to peer 02c53ba06cc3445a8534a84aa76b1c14. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:16.403513 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:16.403807 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:16.404238 21962 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.404560 22111 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.401640 22152 raft_consensus.cc:359] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.420364 22152 raft_consensus.cc:385] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.420735 22152 raft_consensus.cc:740] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.428375 22189 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.433925 22210 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:16.434250 22152 consensus_queue.cc:260] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.436069 22210 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:16.436548 22210 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:16.425371 21962 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:16.439955 22002 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:16.440970 22185 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:16.441419 22152 ts_tablet_manager.cc:1434] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.042s user 0.005s sys 0.004s
I20251023 02:04:16.441444 22185 raft_consensus.cc:697] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:16.442655 22152 tablet_bootstrap.cc:492] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.442589 22185 consensus_queue.cc:237] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.451435 22152 tablet_bootstrap.cc:654] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.434888 22111 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:16.460595 22210 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.470207 22111 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:16.470950 22210 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.471840 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:16.474164 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:16.475209 22077 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:16.468070 22153 tablet_bootstrap.cc:492] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:16.468822 22152 tablet_bootstrap.cc:492] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.476359 22153 ts_tablet_manager.cc:1403] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.116s user 0.044s sys 0.050s
I20251023 02:04:16.467864 22151 tablet_bootstrap.cc:654] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.476028 22191 consensus_queue.cc:1059] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:16.476473 22152 ts_tablet_manager.cc:1403] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.001s
I20251023 02:04:16.475844 22210 raft_consensus.cc:2749] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:16.477227 22191 raft_consensus.cc:3055] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:16.478927 22153 raft_consensus.cc:359] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.481848 22191 raft_consensus.cc:740] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:16.482069 22152 raft_consensus.cc:359] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.482867 22152 raft_consensus.cc:385] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.482668 22191 consensus_queue.cc:260] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.483186 22152 raft_consensus.cc:740] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.483431 22185 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.483870 22185 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.483872 22152 consensus_queue.cc:260] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.481950 22153 raft_consensus.cc:385] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.484992 22153 raft_consensus.cc:740] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.485656 22153 consensus_queue.cc:260] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.486836 22152 ts_tablet_manager.cc:1434] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251023 02:04:16.487327 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.487699 22152 tablet_bootstrap.cc:492] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.487758 22153 ts_tablet_manager.cc:1434] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20251023 02:04:16.487911 21962 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:16.488662 22153 tablet_bootstrap.cc:492] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:16.485992 22185 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.489030 22002 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:16.489777 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:16.490372 22109 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:16.491189 22185 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:16.491539 22185 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:16.491860 22185 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.493294 22153 tablet_bootstrap.cc:654] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.483924 22191 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:16.493753 22152 tablet_bootstrap.cc:654] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.495390 21859 catalog_manager.cc:5649] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:16.483886 22151 tablet_bootstrap.cc:492] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:16.498198 22151 ts_tablet_manager.cc:1403] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.135s user 0.010s sys 0.000s
I20251023 02:04:16.501509 22151 raft_consensus.cc:359] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.502382 22151 raft_consensus.cc:385] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.508824 22152 tablet_bootstrap.cc:492] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.509274 22152 ts_tablet_manager.cc:1403] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.003s
I20251023 02:04:16.505906 22185 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.511817 22153 tablet_bootstrap.cc:492] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:16.512450 22153 ts_tablet_manager.cc:1403] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.003s
I20251023 02:04:16.511831 22152 raft_consensus.cc:359] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.512861 22152 raft_consensus.cc:385] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.513207 22152 raft_consensus.cc:740] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.513916 22152 consensus_queue.cc:260] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.514154 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:16.514853 21962 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.515080 22153 raft_consensus.cc:359] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.515740 22153 raft_consensus.cc:385] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.516013 22153 raft_consensus.cc:740] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.516651 22152 ts_tablet_manager.cc:1434] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251023 02:04:16.516708 22153 consensus_queue.cc:260] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.517786 22152 tablet_bootstrap.cc:492] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.519093 22153 ts_tablet_manager.cc:1434] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251023 02:04:16.520578 22153 tablet_bootstrap.cc:492] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:16.522176 21962 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:16.522969 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:16.523530 22109 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.524394 22152 tablet_bootstrap.cc:654] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.526353 22002 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:16.527195 22189 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:16.527614 22153 tablet_bootstrap.cc:654] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.527664 22189 raft_consensus.cc:697] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:16.534436 22189 consensus_queue.cc:237] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.541723 22185 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.502704 22151 raft_consensus.cc:740] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.543217 22153 tablet_bootstrap.cc:492] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:16.543445 22151 consensus_queue.cc:260] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.543838 22153 ts_tablet_manager.cc:1403] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.009s
I20251023 02:04:16.543871 22109 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:16.547351 22153 raft_consensus.cc:359] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:16.548214 22153 raft_consensus.cc:385] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.548554 22153 raft_consensus.cc:740] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.549396 22153 consensus_queue.cc:260] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:16.553671 22191 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.554126 22191 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.548405 22151 ts_tablet_manager.cc:1434] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.050s user 0.008s sys 0.000s
I20251023 02:04:16.556195 22191 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.557147 22153 ts_tablet_manager.cc:1434] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.013s user 0.010s sys 0.000s
I20251023 02:04:16.557381 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:16.558184 22153 tablet_bootstrap.cc:492] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:16.558857 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.559305 22151 tablet_bootstrap.cc:492] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:16.560357 21928 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:16.561569 22191 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.564934 22151 tablet_bootstrap.cc:654] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.565065 22153 tablet_bootstrap.cc:654] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.564584 21857 catalog_manager.cc:5649] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:16.570565 22152 tablet_bootstrap.cc:492] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.570988 22152 ts_tablet_manager.cc:1403] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.028s
I20251023 02:04:16.575330 22153 tablet_bootstrap.cc:492] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:16.575842 22153 ts_tablet_manager.cc:1403] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20251023 02:04:16.578730 22153 raft_consensus.cc:359] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.578758 22152 raft_consensus.cc:359] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.579586 22153 raft_consensus.cc:385] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.579809 22152 raft_consensus.cc:385] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.580022 22153 raft_consensus.cc:740] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.580235 22152 raft_consensus.cc:740] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.580937 22153 consensus_queue.cc:260] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.581071 22152 consensus_queue.cc:260] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.583453 22152 ts_tablet_manager.cc:1434] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20251023 02:04:16.583957 22153 ts_tablet_manager.cc:1434] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20251023 02:04:16.584309 22152 tablet_bootstrap.cc:492] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.584890 22153 tablet_bootstrap.cc:492] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:16.589787 22153 tablet_bootstrap.cc:654] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.590137 22152 tablet_bootstrap.cc:654] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.594241 22152 tablet_bootstrap.cc:492] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.594658 22152 ts_tablet_manager.cc:1403] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20251023 02:04:16.596596 22151 tablet_bootstrap.cc:492] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:16.597014 22151 ts_tablet_manager.cc:1403] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.006s
I20251023 02:04:16.597375 22152 raft_consensus.cc:359] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.598126 22152 raft_consensus.cc:385] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.598392 22152 raft_consensus.cc:740] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.599052 22152 consensus_queue.cc:260] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.599536 22151 raft_consensus.cc:359] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.600158 22151 raft_consensus.cc:385] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.600401 22151 raft_consensus.cc:740] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.601226 22152 ts_tablet_manager.cc:1434] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251023 02:04:16.600975 22151 consensus_queue.cc:260] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.602912 22151 ts_tablet_manager.cc:1434] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20251023 02:04:16.604051 22152 tablet_bootstrap.cc:492] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.610461 22152 tablet_bootstrap.cc:654] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.615337 22151 tablet_bootstrap.cc:492] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:16.618116 22153 tablet_bootstrap.cc:492] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:16.618633 22153 ts_tablet_manager.cc:1403] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.011s
I20251023 02:04:16.620419 22152 tablet_bootstrap.cc:492] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.620786 22152 ts_tablet_manager.cc:1403] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20251023 02:04:16.621385 22153 raft_consensus.cc:359] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.622066 22153 raft_consensus.cc:385] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.622313 22153 raft_consensus.cc:740] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.622632 22151 tablet_bootstrap.cc:654] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.622838 22152 raft_consensus.cc:359] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.623414 22152 raft_consensus.cc:385] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.622985 22153 consensus_queue.cc:260] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.623659 22152 raft_consensus.cc:740] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.624567 22152 consensus_queue.cc:260] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.625229 22153 ts_tablet_manager.cc:1434] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251023 02:04:16.625768 22210 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.626204 22210 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:16.627997 22210 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:16.628217 22000 consensus_peers.cc:597] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 -> Peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Couldn't send request to peer ca1362b9d9b34073bf35b6f6c9d1159a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:16.629379 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.629539 22152 ts_tablet_manager.cc:1434] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251023 02:04:16.630474 22152 tablet_bootstrap.cc:492] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.632174 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.633155 22036 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 1: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:16.636018 22152 tablet_bootstrap.cc:654] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.637724 22153 tablet_bootstrap.cc:492] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:16.645229 22153 tablet_bootstrap.cc:654] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.645998 22210 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.646481 22210 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.647193 22077 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:16.648298 22202 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.649791 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:16.651156 22076 leader_election.cc:343] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.654404 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.654621 22151 tablet_bootstrap.cc:492] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:16.655544 22212 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.655761 22151 ts_tablet_manager.cc:1403] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.041s user 0.020s sys 0.005s
I20251023 02:04:16.656003 22212 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:16.657828 22077 leader_election.cc:343] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.658278 22077 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:16.659597 22151 raft_consensus.cc:359] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.660532 22151 raft_consensus.cc:385] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.660808 22159 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.660948 22151 raft_consensus.cc:740] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.661916 22212 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.648365 22210 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.662012 22151 consensus_queue.cc:260] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.665149 22151 ts_tablet_manager.cc:1434] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20251023 02:04:16.665233 22202 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.666002 22202 consensus_queue.cc:1048] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:16.666188 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.666882 21962 raft_consensus.cc:2393] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 1: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:16.668601 22191 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.669083 22191 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.671350 22191 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.675052 22151 tablet_bootstrap.cc:492] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:16.674955 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:16.676124 21928 leader_election.cc:343] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.676581 22152 tablet_bootstrap.cc:492] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.677150 22152 ts_tablet_manager.cc:1403] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.015s
I20251023 02:04:16.682202 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:16.683727 21925 leader_election.cc:343] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.684141 21925 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:16.685079 22171 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.684330 22152 raft_consensus.cc:359] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.685783 22152 raft_consensus.cc:385] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.686110 22152 raft_consensus.cc:740] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.686558 22151 tablet_bootstrap.cc:654] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.686832 22152 consensus_queue.cc:260] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.689023 22152 ts_tablet_manager.cc:1434] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.011s user 0.002s sys 0.002s
I20251023 02:04:16.690092 22152 tablet_bootstrap.cc:492] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:16.692544 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:16.693742 21908 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251023 02:04:16.694377 22077 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:16.695531 22243 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.699699 22153 tablet_bootstrap.cc:492] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:16.701858 22153 ts_tablet_manager.cc:1403] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.064s user 0.028s sys 0.015s
I20251023 02:04:16.702257 22151 tablet_bootstrap.cc:492] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:16.702724 22151 ts_tablet_manager.cc:1403] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.003s
I20251023 02:04:16.705251 22151 raft_consensus.cc:359] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.705909 22151 raft_consensus.cc:385] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.706202 22151 raft_consensus.cc:740] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.708534 22153 raft_consensus.cc:359] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.709233 22153 raft_consensus.cc:385] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.709633 22153 raft_consensus.cc:740] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.710361 22153 consensus_queue.cc:260] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.711545 22191 consensus_queue.cc:1048] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:16.705677 22152 tablet_bootstrap.cc:654] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.712215 22036 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.713351 22153 ts_tablet_manager.cc:1434] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20251023 02:04:16.706887 22151 consensus_queue.cc:260] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.715807 22185 consensus_queue.cc:1048] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:16.716646 22153 tablet_bootstrap.cc:492] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:16.720046 22157 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.720741 22157 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.723971 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.724382 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:16.724640 21961 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:16.725126 22151 ts_tablet_manager.cc:1434] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20251023 02:04:16.722904 22157 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.725011 22111 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:16.726120 22151 tablet_bootstrap.cc:492] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:16.726518 22000 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:16.727324 22157 raft_consensus.cc:2804] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:16.727659 22157 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:16.727977 22157 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.735590 22157 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.738866 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:16.740084 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:16.745122 22111 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.746176 21961 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.739740 22157 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.747663 22189 consensus_queue.cc:1048] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:16.751920 22153 tablet_bootstrap.cc:654] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.760361 22111 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:16.761567 22243 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.761656 22000 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:16.762547 22157 raft_consensus.cc:2804] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:16.760908 22185 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.763677 22185 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.766304 22185 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.763182 22157 raft_consensus.cc:697] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:16.770217 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.771684 22157 consensus_queue.cc:237] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.770352 22210 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.773125 22210 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.774010 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:16.775143 22202 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.776104 22210 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.775296 22189 consensus_queue.cc:1048] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251023 02:04:16.777261 22002 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:16.778432 22185 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:16.779412 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.779533 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.780289 21959 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:16.780593 22036 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:16.774772 22243 consensus_queue.cc:1048] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20251023 02:04:16.781688 22076 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:16.776187 22202 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.792222 22210 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:16.798884 22210 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:16.799484 22210 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.795787 22191 consensus_queue.cc:1048] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:16.804013 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.804858 22037 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:16.798167 22202 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.806358 22077 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:16.810513 22202 raft_consensus.cc:2804] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:16.810963 22202 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:16.811352 22202 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.792472 21961 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:16.829849 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:16.831615 22076 leader_election.cc:343] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.839581 22191 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.840097 22191 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.842041 22191 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.842805 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.843588 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:16.843849 22202 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:16.845005 21925 leader_election.cc:343] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.849421 22153 tablet_bootstrap.cc:492] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:16.850008 22153 ts_tablet_manager.cc:1403] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.134s user 0.018s sys 0.012s
I20251023 02:04:16.847755 21860 catalog_manager.cc:5649] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
W20251023 02:04:16.851500 21928 leader_election.cc:343] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.851954 21928 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:16.851856 22210 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.852996 22191 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.853338 22153 raft_consensus.cc:359] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.853992 22153 raft_consensus.cc:385] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.854244 22153 raft_consensus.cc:740] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.854446 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:16.854961 21959 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.854951 22153 consensus_queue.cc:260] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.859928 22210 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.860677 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:16.864229 22037 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.864403 22191 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.864871 22191 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.867209 22191 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.868770 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:16.870221 21925 leader_election.cc:343] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.874193 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.875916 22202 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.860869 22157 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.877530 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:16.878253 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:16.878413 21928 leader_election.cc:343] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.881084 22185 consensus_queue.cc:1048] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:16.886885 22157 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.887162 21928 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:16.887738 22076 leader_election.cc:343] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.887893 22191 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.888309 22034 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.882181 22151 tablet_bootstrap.cc:654] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:16.885987 22153 ts_tablet_manager.cc:1434] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20251023 02:04:16.889776 22153 tablet_bootstrap.cc:492] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:16.890954 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.891561 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:16.893203 22037 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:16.891633 21962 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:16.893860 22152 tablet_bootstrap.cc:492] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:16.891896 22157 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.895857 22152 ts_tablet_manager.cc:1403] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.206s user 0.015s sys 0.011s
W20251023 02:04:16.895641 22000 leader_election.cc:343] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.894953 21959 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:16.896982 22002 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:16.897729 22076 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:16.898119 22185 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:16.898475 22185 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:16.898810 22185 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.901124 22152 raft_consensus.cc:359] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.901870 22210 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:16.902072 22152 raft_consensus.cc:385] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:16.902500 22152 raft_consensus.cc:740] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:16.903684 22243 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.904361 22243 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.903736 22152 consensus_queue.cc:260] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.905474 22202 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.906483 22243 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.906068 22202 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.903499 22210 raft_consensus.cc:697] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:16.929395 22191 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.930840 22034 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:16.931051 22191 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.933864 22152 ts_tablet_manager.cc:1434] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20251023 02:04:16.934698 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.936676 22191 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:16.938584 22202 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:16.938690 22077 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:16.939846 22210 consensus_queue.cc:237] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:16.940549 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.940708 22243 raft_consensus.cc:2804] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:16.941762 22243 raft_consensus.cc:697] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:16.943439 21959 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:16.945816 22185 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:16.946210 22077 leader_election.cc:343] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.947716 22185 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:16.949534 22076 leader_election.cc:343] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.949710 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:16.949957 22076 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:16.950639 22210 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:16.950788 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:16.942950 22243 consensus_queue.cc:237] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:16.955054 22000 leader_election.cc:343] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.952324 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:16.956089 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:17.034544 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 79560.7us
W20251023 02:04:16.952814 22077 leader_election.cc:343] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:16.952311 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:17.036397 22076 leader_election.cc:343] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.036834 22034 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:16.978528 22158 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:16.952390 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:17.026657 22152 tablet_bootstrap.cc:492] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:17.041560 21928 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:17.036319 21962 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.045377 22151 tablet_bootstrap.cc:492] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:17.045828 22151 ts_tablet_manager.cc:1403] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.320s user 0.021s sys 0.007s
I20251023 02:04:17.039527 22191 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.047236 22191 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.049024 22191 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.044075 22192 raft_consensus.cc:2804] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:17.049515 22192 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:17.049886 22192 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.050304 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:17.048620 22151 raft_consensus.cc:359] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.051465 22151 raft_consensus.cc:385] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.051668 22151 raft_consensus.cc:740] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.052100 22151 consensus_queue.cc:260] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:17.052812 21925 leader_election.cc:343] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.054898 22151 ts_tablet_manager.cc:1434] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20251023 02:04:17.055502 22153 tablet_bootstrap.cc:654] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.056568 22151 tablet_bootstrap.cc:492] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:17.037559 22158 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.043749 22111 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:17.040428 22077 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:17.058857 22185 consensus_queue.cc:1048] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:17.060617 22191 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.063290 22191 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.065142 22191 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.065524 22210 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:17.067786 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:17.069795 22158 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:17.069938 21925 leader_election.cc:343] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.071447 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:17.072710 21925 leader_election.cc:343] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.072957 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:17.074872 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:17.075475 22037 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
W20251023 02:04:17.077386 21928 leader_election.cc:343] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.077844 21928 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:17.078500 22191 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:17.073405 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:17.065534 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 112809us
I20251023 02:04:17.079453 21928 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:17.091568 22191 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:17.092080 22191 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:17.092507 22191 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.110927 22111 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.106446 22192 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.151898 22192 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.153363 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:17.154078 22109 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.162119 22203 consensus_queue.cc:1048] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20251023 02:04:17.198678 22152 tablet_bootstrap.cc:654] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
W20251023 02:04:17.200495 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 121148us
W20251023 02:04:17.207183 21928 leader_election.cc:343] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.208738 22153 tablet_bootstrap.cc:492] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:17.208503 22151 tablet_bootstrap.cc:654] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.209175 22153 ts_tablet_manager.cc:1403] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.320s user 0.019s sys 0.000s
I20251023 02:04:17.209980 21928 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:17.214998 22192 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:17.212522 22203 consensus_queue.cc:1048] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251023 02:04:17.219606 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:17.221660 22034 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.225409 21962 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:17.227093 22002 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:17.228309 22203 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:17.228760 22203 raft_consensus.cc:697] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:17.229036 22034 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:17.229624 22203 consensus_queue.cc:237] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.230263 21928 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:17.229194 22153 raft_consensus.cc:359] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.231241 22153 raft_consensus.cc:385] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.231635 22153 raft_consensus.cc:740] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.233023 22192 raft_consensus.cc:2804] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:17.233492 22192 raft_consensus.cc:697] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:17.234603 22192 consensus_queue.cc:237] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.235641 22153 consensus_queue.cc:260] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.235700 22191 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.238107 22153 ts_tablet_manager.cc:1434] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20251023 02:04:17.238996 22153 tablet_bootstrap.cc:492] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:17.240232 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:17.235338 22109 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:17.247272 22034 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.249126 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:04:17.250516 21925 leader_election.cc:343] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.247413 22191 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.259965 22034 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:17.261530 21928 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:17.267792 22191 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:17.268451 22191 raft_consensus.cc:697] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:17.271504 22152 tablet_bootstrap.cc:492] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:17.271847 22152 ts_tablet_manager.cc:1403] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.246s user 0.015s sys 0.000s
I20251023 02:04:17.269362 22191 consensus_queue.cc:237] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.275918 21860 catalog_manager.cc:5649] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:17.285705 21857 catalog_manager.cc:5649] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:17.287170 21858 catalog_manager.cc:5649] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:17.290095 22152 raft_consensus.cc:359] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.289829 21858 catalog_manager.cc:5649] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:17.294955 22202 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.294932 22203 consensus_queue.cc:1048] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:17.295539 22202 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.293280 22152 raft_consensus.cc:385] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.297811 22152 raft_consensus.cc:740] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.300355 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:17.300985 21959 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:17.301807 22202 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:17.302323 22076 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:17.302359 22185 consensus_queue.cc:1048] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:17.298334 22152 consensus_queue.cc:260] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.304147 22151 tablet_bootstrap.cc:492] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:17.304555 22151 ts_tablet_manager.cc:1403] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.249s user 0.008s sys 0.009s
I20251023 02:04:17.305334 22202 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:17.306499 22202 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:17.306874 22202 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.314673 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:17.318704 22077 leader_election.cc:343] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.323149 22153 tablet_bootstrap.cc:654] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.332078 21857 catalog_manager.cc:5649] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:17.338800 22152 ts_tablet_manager.cc:1434] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.067s user 0.007s sys 0.000s
I20251023 02:04:17.340080 22152 tablet_bootstrap.cc:492] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:17.343987 22202 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.347733 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:17.348287 21961 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.352662 22202 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:17.355355 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:17.361608 22077 leader_election.cc:343] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.362782 21961 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:17.363885 22076 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:17.365556 22202 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:17.364604 22151 raft_consensus.cc:359] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.366173 22151 raft_consensus.cc:385] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.366459 22151 raft_consensus.cc:740] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.367079 22151 consensus_queue.cc:260] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.367437 22202 raft_consensus.cc:697] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:17.371089 22202 consensus_queue.cc:237] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.374298 22152 tablet_bootstrap.cc:654] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.379624 22151 ts_tablet_manager.cc:1434] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.075s user 0.007s sys 0.003s
I20251023 02:04:17.380517 22151 tablet_bootstrap.cc:492] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:17.389873 22153 tablet_bootstrap.cc:492] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:17.390344 22153 ts_tablet_manager.cc:1403] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.152s user 0.018s sys 0.008s
I20251023 02:04:17.393471 22153 raft_consensus.cc:359] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.394135 22153 raft_consensus.cc:385] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.394402 22153 raft_consensus.cc:740] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.395040 22153 consensus_queue.cc:260] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.397264 22151 tablet_bootstrap.cc:654] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.405340 22152 tablet_bootstrap.cc:492] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:17.405966 22152 ts_tablet_manager.cc:1403] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.066s user 0.026s sys 0.004s
I20251023 02:04:17.406142 22153 ts_tablet_manager.cc:1434] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20251023 02:04:17.407037 22153 tablet_bootstrap.cc:492] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:17.407831 21860 catalog_manager.cc:5649] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:17.409232 22152 raft_consensus.cc:359] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.410137 22152 raft_consensus.cc:385] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.410476 22152 raft_consensus.cc:740] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.411240 22152 consensus_queue.cc:260] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.414163 22152 ts_tablet_manager.cc:1434] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20251023 02:04:17.415544 22152 tablet_bootstrap.cc:492] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:17.416477 22153 tablet_bootstrap.cc:654] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.422322 22151 tablet_bootstrap.cc:492] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:17.422855 22152 tablet_bootstrap.cc:654] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.423395 22151 ts_tablet_manager.cc:1403] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.043s user 0.023s sys 0.009s
I20251023 02:04:17.426635 22151 raft_consensus.cc:359] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.427497 22151 raft_consensus.cc:385] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.428658 22151 raft_consensus.cc:740] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.429436 22151 consensus_queue.cc:260] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.432557 22151 ts_tablet_manager.cc:1434] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20251023 02:04:17.430480 22153 tablet_bootstrap.cc:492] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:17.433662 22151 tablet_bootstrap.cc:492] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:17.434090 22153 ts_tablet_manager.cc:1403] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.003s
I20251023 02:04:17.437615 22153 raft_consensus.cc:359] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.438334 22153 raft_consensus.cc:385] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.438741 22153 raft_consensus.cc:740] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.441290 22151 tablet_bootstrap.cc:654] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.439620 22153 consensus_queue.cc:260] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.443076 22152 tablet_bootstrap.cc:492] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:17.443506 22152 ts_tablet_manager.cc:1403] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.005s
I20251023 02:04:17.443558 22153 ts_tablet_manager.cc:1434] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251023 02:04:17.444864 22153 tablet_bootstrap.cc:492] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:17.446851 22152 raft_consensus.cc:359] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.447700 22152 raft_consensus.cc:385] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.448045 22152 raft_consensus.cc:740] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.448907 22152 consensus_queue.cc:260] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.451189 22153 tablet_bootstrap.cc:654] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.457247 22202 consensus_queue.cc:1048] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:17.466491 22152 ts_tablet_manager.cc:1434] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20251023 02:04:17.467698 22152 tablet_bootstrap.cc:492] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:17.470531 22151 tablet_bootstrap.cc:492] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:17.470949 22151 ts_tablet_manager.cc:1403] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.024s
I20251023 02:04:17.474460 22152 tablet_bootstrap.cc:654] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.475179 21962 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.475888 22151 raft_consensus.cc:359] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.476576 22151 raft_consensus.cc:385] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.476887 22151 raft_consensus.cc:740] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.478390 22151 consensus_queue.cc:260] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.480722 22151 ts_tablet_manager.cc:1434] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
I20251023 02:04:17.482985 22152 tablet_bootstrap.cc:492] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:17.483408 22152 ts_tablet_manager.cc:1403] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.002s
I20251023 02:04:17.484984 22152 raft_consensus.cc:359] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.485386 22152 raft_consensus.cc:385] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.485546 22152 raft_consensus.cc:740] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.486039 22152 consensus_queue.cc:260] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.487496 22152 ts_tablet_manager.cc:1434] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251023 02:04:17.488095 22152 tablet_bootstrap.cc:492] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:17.492483 22152 tablet_bootstrap.cc:654] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.492645 22151 tablet_bootstrap.cc:492] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:17.504885 22151 tablet_bootstrap.cc:654] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.505481 22210 consensus_queue.cc:1048] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:17.526702 22151 tablet_bootstrap.cc:492] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:17.527159 22151 ts_tablet_manager.cc:1403] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.001s
I20251023 02:04:17.533694 22151 raft_consensus.cc:359] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.534375 22151 raft_consensus.cc:385] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.534770 22151 raft_consensus.cc:740] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.536732 22151 consensus_queue.cc:260] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.554565 22151 ts_tablet_manager.cc:1434] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.027s user 0.006s sys 0.003s
I20251023 02:04:17.555472 22151 tablet_bootstrap.cc:492] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:17.561369 22151 tablet_bootstrap.cc:654] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.573565 22202 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:17.574661 22152 tablet_bootstrap.cc:492] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:17.575101 22152 ts_tablet_manager.cc:1403] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.087s user 0.027s sys 0.052s
I20251023 02:04:17.575242 22153 tablet_bootstrap.cc:492] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:17.569187 22171 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.575683 22153 ts_tablet_manager.cc:1403] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.131s user 0.034s sys 0.055s
I20251023 02:04:17.575949 22171 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.577922 22171 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.578040 22153 raft_consensus.cc:359] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.578680 22153 raft_consensus.cc:385] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.578506 22152 raft_consensus.cc:359] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.579020 22153 raft_consensus.cc:740] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.579416 22152 raft_consensus.cc:385] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.579449 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:17.579931 22152 raft_consensus.cc:740] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.579805 22153 consensus_queue.cc:260] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:17.580883 21928 leader_election.cc:343] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.580788 22152 consensus_queue.cc:260] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.583905 22152 ts_tablet_manager.cc:1434] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20251023 02:04:17.584982 22152 tablet_bootstrap.cc:492] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:17.585331 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:17.585924 22110 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:17.589493 21925 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:17.590504 22171 raft_consensus.cc:2804] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:17.590873 22171 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:17.591202 22171 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.597128 22171 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.598850 22171 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.599152 22153 ts_tablet_manager.cc:1434] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20251023 02:04:17.599625 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:17.600010 22153 tablet_bootstrap.cc:492] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:17.600026 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:17.600651 22110 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:17.601171 21928 leader_election.cc:343] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.606403 22171 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.606851 22171 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.608639 22171 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.610167 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:17.610625 22034 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:17.610818 22110 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:17.611581 21928 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:17.611974 21925 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:17.612402 22171 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:17.612675 22191 raft_consensus.cc:2804] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:17.612722 22171 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:17.613106 22171 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.613178 22191 raft_consensus.cc:697] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:17.613775 22152 tablet_bootstrap.cc:654] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.614042 22191 consensus_queue.cc:237] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.618892 22151 tablet_bootstrap.cc:492] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:17.619534 22151 ts_tablet_manager.cc:1403] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.064s user 0.016s sys 0.019s
I20251023 02:04:17.619626 22153 tablet_bootstrap.cc:654] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.622011 22171 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.622920 22151 raft_consensus.cc:359] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.623797 22151 raft_consensus.cc:385] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.613524 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:17.624176 22151 raft_consensus.cc:740] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.624461 22110 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:17.624974 22202 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:17.625989 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:17.626593 22034 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.624815 22151 consensus_queue.cc:260] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.636518 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:17.637034 22171 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.637086 22110 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.638020 22243 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.638609 22243 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.641074 22243 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:17.643903 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:17.645368 22151 ts_tablet_manager.cc:1434] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.026s user 0.007s sys 0.001s
W20251023 02:04:17.645668 22077 leader_election.cc:343] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.648165 22151 tablet_bootstrap.cc:492] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:17.646767 22034 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:17.644340 22110 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:17.650636 21928 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:17.651530 22191 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:17.651999 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:17.653882 22202 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.654327 22202 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.656134 22202 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:17.652187 22191 raft_consensus.cc:697] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:04:17.659158 22076 leader_election.cc:343] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.659008 22191 consensus_queue.cc:237] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.659581 22076 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:17.656229 21860 catalog_manager.cc:5649] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:17.661901 22192 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251023 02:04:17.663432 22202 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:17.652246 22171 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.664295 22171 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.663903 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:17.669802 22037 raft_consensus.cc:2393] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 1: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:17.671692 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:17.691339 22153 tablet_bootstrap.cc:492] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:17.691963 22153 ts_tablet_manager.cc:1403] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.007s
I20251023 02:04:17.700811 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:17.705101 22192 consensus_queue.cc:1048] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:17.705472 22110 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:17.710111 22171 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.710368 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:17.712031 22036 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:17.713779 22152 tablet_bootstrap.cc:492] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:17.714242 22152 ts_tablet_manager.cc:1403] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.129s user 0.020s sys 0.024s
I20251023 02:04:17.709381 22151 tablet_bootstrap.cc:654] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.717464 21928 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:17.720050 22192 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:17.720487 22192 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:17.720871 22192 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:17.722709 21908 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20251023 02:04:17.729713 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 57827us
I20251023 02:04:17.730733 22191 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:17.733907 22153 raft_consensus.cc:359] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.734544 22076 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:17.734817 22153 raft_consensus.cc:385] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.735332 22202 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.738076 22192 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.764835 22192 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.749290 22157 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:17.750680 22152 raft_consensus.cc:359] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.751595 22153 raft_consensus.cc:740] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.760468 21860 catalog_manager.cc:5649] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:17.771862 22153 consensus_queue.cc:260] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.766170 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:17.772986 22111 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.744292 22202 raft_consensus.cc:2749] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:17.774016 22153 ts_tablet_manager.cc:1434] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.082s user 0.010s sys 0.001s
I20251023 02:04:17.774930 22153 tablet_bootstrap.cc:492] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:17.775797 22152 raft_consensus.cc:385] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.776068 22152 raft_consensus.cc:740] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.766587 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:17.776633 22152 consensus_queue.cc:260] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.783682 22152 ts_tablet_manager.cc:1434] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.069s user 0.004s sys 0.000s
I20251023 02:04:17.784487 22152 tablet_bootstrap.cc:492] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:17.784984 22192 consensus_queue.cc:1048] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:17.789924 22152 tablet_bootstrap.cc:654] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.790551 22189 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.791038 22189 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.793035 22189 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.794915 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:17.798432 22108 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:17.800216 22000 leader_election.cc:343] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.800741 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:17.802067 21961 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:17.803510 22002 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:17.766994 22151 tablet_bootstrap.cc:492] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:17.780089 22037 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.781519 22153 tablet_bootstrap.cc:654] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.806875 22151 ts_tablet_manager.cc:1403] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.159s user 0.016s sys 0.017s
I20251023 02:04:17.779985 22111 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:17.811618 22151 raft_consensus.cc:359] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.812212 21925 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:17.804915 22189 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:17.812247 22151 raft_consensus.cc:385] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.813009 22192 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:17.813042 22151 raft_consensus.cc:740] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.814071 22192 raft_consensus.cc:697] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:17.816128 22151 consensus_queue.cc:260] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.815510 22192 consensus_queue.cc:237] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.818490 22151 ts_tablet_manager.cc:1434] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20251023 02:04:17.819990 22151 tablet_bootstrap.cc:492] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:17.812906 22189 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:17.820487 22189 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.881217 22158 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.887140 22158 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.876530 22185 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.914057 22185 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.910508 22203 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.919860 22189 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.922672 22153 tablet_bootstrap.cc:492] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:17.923120 22153 ts_tablet_manager.cc:1403] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.148s user 0.009s sys 0.012s
I20251023 02:04:17.923408 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:17.925223 21962 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.925127 22202 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:17.926851 22153 raft_consensus.cc:359] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.927490 22153 raft_consensus.cc:385] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.927757 22153 raft_consensus.cc:740] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.928360 22153 consensus_queue.cc:260] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.933835 22189 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.934041 22153 ts_tablet_manager.cc:1434] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20251023 02:04:17.935427 22153 tablet_bootstrap.cc:492] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:17.936523 21860 catalog_manager.cc:5649] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:17.937704 22037 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:17.948459 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:17.949170 22035 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:17.948601 21962 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:17.949762 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:17.951632 22185 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.952270 22152 tablet_bootstrap.cc:492] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:17.952831 22152 ts_tablet_manager.cc:1403] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.169s user 0.036s sys 0.052s
I20251023 02:04:17.931002 22157 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:17.955156 21928 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:17.930083 22203 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.939399 22158 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.957412 22192 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:17.958431 22192 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:17.958755 22192 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:17.960907 22243 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:17.961707 22203 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.961547 22243 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.957165 22157 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.962980 22151 tablet_bootstrap.cc:654] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:17.970059 22152 raft_consensus.cc:359] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.970557 22152 raft_consensus.cc:385] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:17.970758 22152 raft_consensus.cc:740] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:17.971242 22152 consensus_queue.cc:260] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.968905 22157 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:17.973745 22243 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:17.969487 22000 leader_election.cc:343] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.977890 22203 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:17.980445 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:17.981792 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:17.981904 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:17.982533 22110 raft_consensus.cc:2393] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 1: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:17.983144 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:17.983247 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:17.983563 22152 ts_tablet_manager.cc:1434] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.030s user 0.004s sys 0.004s
I20251023 02:04:17.983770 22107 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:17.984508 22152 tablet_bootstrap.cc:492] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:17.984723 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:17.985441 22109 raft_consensus.cc:2393] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
W20251023 02:04:17.975826 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 50133.5us
I20251023 02:04:17.990756 22076 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:17.991053 21925 leader_election.cc:343] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:17.989166 22000 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:17.998451 22151 tablet_bootstrap.cc:492] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:17.998991 22151 ts_tablet_manager.cc:1403] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.179s user 0.014s sys 0.000s
I20251023 02:04:17.987414 22192 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:17.999668 22303 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:18.000191 22303 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:18.000450 22303 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.003583 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:18.002876 22192 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.004496 22035 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.005215 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.026036 22035 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:18.027261 21928 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:18.027176 22151 raft_consensus.cc:359] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.028865 22151 raft_consensus.cc:385] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.029078 22151 raft_consensus.cc:740] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
W20251023 02:04:18.028527 22077 consensus_peers.cc:597] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Couldn't send request to peer 818907d63f1f4d84ba267ebe6ee837e0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:18.029507 22192 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:18.029769 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.030030 22192 raft_consensus.cc:697] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:18.029800 22151 consensus_queue.cc:260] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.030469 21962 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:18.030861 22192 consensus_queue.cc:237] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.032282 22151 ts_tablet_manager.cc:1434] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.033s user 0.019s sys 0.002s
I20251023 02:04:18.028157 22153 tablet_bootstrap.cc:654] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.036120 22151 tablet_bootstrap.cc:492] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:18.037698 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.038314 21959 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:18.044091 22002 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:18.046898 22189 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:18.047171 22152 tablet_bootstrap.cc:654] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.047392 22189 raft_consensus.cc:697] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:18.047362 22210 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:18.043848 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:18.052381 22189 consensus_queue.cc:237] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.055245 22002 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:18.057020 21925 leader_election.cc:343] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.058019 22203 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.058681 22002 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:18.059880 22185 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:18.060223 22185 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:18.060546 22185 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:18.061421 21860 catalog_manager.cc:5649] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:18.064836 22203 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:18.068166 22303 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.068723 22185 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.071810 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:18.072397 22106 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.073787 22151 tablet_bootstrap.cc:654] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.073688 22158 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:18.075012 22158 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:18.079473 22210 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.079937 22210 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.083561 22210 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:18.085269 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.087963 21962 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:18.089082 22076 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:18.089990 22210 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:18.090402 22210 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:18.090723 22210 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.094321 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:18.096840 22077 leader_election.cc:343] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.100788 22185 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.113767 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:18.109721 22303 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.124603 22192 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.131232 22191 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:18.132880 21857 catalog_manager.cc:5649] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:18.135038 22106 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:18.137435 22210 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.136749 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:18.147589 22171 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20251023 02:04:18.137251 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:18.135165 22192 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.136822 22000 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:18.179438 22152 tablet_bootstrap.cc:492] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:18.186627 22152 ts_tablet_manager.cc:1403] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.202s user 0.009s sys 0.033s
I20251023 02:04:18.182591 22153 tablet_bootstrap.cc:492] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:18.187847 22153 ts_tablet_manager.cc:1403] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.253s user 0.006s sys 0.038s
I20251023 02:04:18.137413 22109 raft_consensus.cc:2393] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:18.191463 22152 raft_consensus.cc:359] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.192075 22151 tablet_bootstrap.cc:492] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:18.192521 22151 ts_tablet_manager.cc:1403] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.157s user 0.014s sys 0.008s
I20251023 02:04:18.192097 22152 raft_consensus.cc:385] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.193387 22152 raft_consensus.cc:740] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.193945 22152 consensus_queue.cc:260] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.194386 22153 raft_consensus.cc:359] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.177435 21962 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:18.185547 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 48822.9us
I20251023 02:04:18.186084 22189 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:18.195708 22153 raft_consensus.cc:385] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.197522 22152 ts_tablet_manager.cc:1434] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20251023 02:04:18.198060 22153 raft_consensus.cc:740] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.181335 22210 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:18.198860 22153 consensus_queue.cc:260] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.202138 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:18.204595 22152 tablet_bootstrap.cc:492] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:18.197391 22192 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.208493 22153 ts_tablet_manager.cc:1434] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20251023 02:04:18.209381 22153 tablet_bootstrap.cc:492] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:18.208638 22151 raft_consensus.cc:359] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.210189 22151 raft_consensus.cc:385] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.210469 22151 raft_consensus.cc:740] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.201265 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:18.210958 22151 consensus_queue.cc:260] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.208063 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:18.205313 22189 raft_consensus.cc:697] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:04:18.215636 21928 leader_election.cc:343] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.215838 21961 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.215808 22189 consensus_queue.cc:237] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:18.216899 21925 leader_election.cc:343] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.217367 21925 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:18.219249 22158 raft_consensus.cc:2749] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:18.227519 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:18.229252 22077 leader_election.cc:343] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.233502 22158 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:18.236253 22151 ts_tablet_manager.cc:1434] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.043s user 0.003s sys 0.002s
I20251023 02:04:18.236940 22151 tablet_bootstrap.cc:492] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:18.249809 21962 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:18.246692 21959 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:18.254343 22002 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:18.256037 21961 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:18.256788 22185 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:18.257287 22185 raft_consensus.cc:697] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:18.257577 22076 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:18.258111 22185 consensus_queue.cc:237] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.258973 22210 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:18.259298 22153 tablet_bootstrap.cc:654] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.260074 22210 raft_consensus.cc:697] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:18.270581 22210 consensus_queue.cc:237] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.273213 21860 catalog_manager.cc:5649] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:18.279301 22151 tablet_bootstrap.cc:654] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.288977 22152 tablet_bootstrap.cc:654] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.290200 21860 catalog_manager.cc:5649] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:18.294343 21858 catalog_manager.cc:5649] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 2, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 2 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:18.313792 22171 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:18.327564 22151 tablet_bootstrap.cc:492] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:18.328007 22151 ts_tablet_manager.cc:1403] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.005s
I20251023 02:04:18.330420 22151 raft_consensus.cc:359] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.331152 22151 raft_consensus.cc:385] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.331434 22151 raft_consensus.cc:740] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.331951 22153 tablet_bootstrap.cc:492] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:18.332427 22152 tablet_bootstrap.cc:492] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:18.332051 22151 consensus_queue.cc:260] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.332625 22153 ts_tablet_manager.cc:1403] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.031s
I20251023 02:04:18.332906 22152 ts_tablet_manager.cc:1403] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.128s user 0.011s sys 0.000s
I20251023 02:04:18.333791 22151 ts_tablet_manager.cc:1434] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251023 02:04:18.334618 22151 tablet_bootstrap.cc:492] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:18.335435 22153 raft_consensus.cc:359] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.336127 22153 raft_consensus.cc:385] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.336551 22153 raft_consensus.cc:740] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.335201 22152 raft_consensus.cc:359] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.337473 22152 raft_consensus.cc:385] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.337891 22152 raft_consensus.cc:740] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.338716 22152 consensus_queue.cc:260] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.340062 22151 tablet_bootstrap.cc:654] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.339908 22153 consensus_queue.cc:260] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.342648 22152 ts_tablet_manager.cc:1434] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251023 02:04:18.347316 22152 tablet_bootstrap.cc:492] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:18.350708 22153 ts_tablet_manager.cc:1434] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.018s user 0.001s sys 0.005s
I20251023 02:04:18.351687 22153 tablet_bootstrap.cc:492] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:18.357357 22153 tablet_bootstrap.cc:654] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.357307 22152 tablet_bootstrap.cc:654] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.368178 22152 tablet_bootstrap.cc:492] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:18.368813 22152 ts_tablet_manager.cc:1403] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.009s
I20251023 02:04:18.370483 22153 tablet_bootstrap.cc:492] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:18.371368 22185 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.371824 22185 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.370921 22153 ts_tablet_manager.cc:1403] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.003s
I20251023 02:04:18.372763 22152 raft_consensus.cc:359] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.373853 22185 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.374881 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:18.376101 22152 raft_consensus.cc:385] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.376031 22153 raft_consensus.cc:359] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.375617 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.376813 22153 raft_consensus.cc:385] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.375222 22151 tablet_bootstrap.cc:492] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:18.377110 22153 raft_consensus.cc:740] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.377650 22151 ts_tablet_manager.cc:1403] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.008s
I20251023 02:04:18.376509 22152 raft_consensus.cc:740] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.377912 22305 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.377866 22153 consensus_queue.cc:260] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.378522 22305 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.380115 22151 raft_consensus.cc:359] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.380654 22305 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:18.380781 22151 raft_consensus.cc:385] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.380492 22152 consensus_queue.cc:260] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.381096 22151 raft_consensus.cc:740] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.381732 22151 consensus_queue.cc:260] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.380149 22153 ts_tablet_manager.cc:1434] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251023 02:04:18.382579 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:18.383392 22034 raft_consensus.cc:2393] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 1: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:18.383450 22152 ts_tablet_manager.cc:1434] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20251023 02:04:18.383575 22153 tablet_bootstrap.cc:492] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:18.383927 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.384428 22152 tablet_bootstrap.cc:492] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:18.389757 22153 tablet_bootstrap.cc:654] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.389988 22152 tablet_bootstrap.cc:654] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.392717 22076 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:18.397717 22210 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.378516 22002 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:18.406654 22185 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:18.421250 22305 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.422060 22305 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.425410 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.426326 22210 raft_consensus.cc:2749] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:18.426470 21961 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:18.425338 22151 ts_tablet_manager.cc:1434] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20251023 02:04:18.428107 22076 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:18.429250 22202 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:18.429695 22202 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:18.430088 22202 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:18.432965 22151 tablet_bootstrap.cc:492] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:18.443466 22305 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:18.446107 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:18.446524 22202 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.454972 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:18.455817 21961 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:18.458195 22151 tablet_bootstrap.cc:654] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.464882 22152 tablet_bootstrap.cc:492] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:18.465307 22152 ts_tablet_manager.cc:1403] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.081s user 0.029s sys 0.050s
I20251023 02:04:18.466789 22202 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:18.467463 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:18.467656 21961 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:18.467729 22152 raft_consensus.cc:359] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.469520 22152 raft_consensus.cc:385] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.469816 22152 raft_consensus.cc:740] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.470144 22153 tablet_bootstrap.cc:492] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:18.470271 22076 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:18.470403 22152 consensus_queue.cc:260] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.470743 22153 ts_tablet_manager.cc:1403] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.087s user 0.037s sys 0.048s
I20251023 02:04:18.472098 22152 ts_tablet_manager.cc:1434] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251023 02:04:18.472582 22202 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:18.473044 22202 raft_consensus.cc:697] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:18.473153 22153 raft_consensus.cc:359] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.473800 22153 raft_consensus.cc:385] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.474053 22153 raft_consensus.cc:740] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.473901 22202 consensus_queue.cc:237] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.474573 22153 consensus_queue.cc:260] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.476451 22153 ts_tablet_manager.cc:1434] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251023 02:04:18.476464 22151 tablet_bootstrap.cc:492] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:18.476974 22151 ts_tablet_manager.cc:1403] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.044s user 0.018s sys 0.000s
I20251023 02:04:18.477262 22153 tablet_bootstrap.cc:492] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:18.477403 22152 tablet_bootstrap.cc:492] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:18.479295 22151 raft_consensus.cc:359] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.479915 22151 raft_consensus.cc:385] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.480216 22151 raft_consensus.cc:740] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.480840 22151 consensus_queue.cc:260] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.482887 22152 tablet_bootstrap.cc:654] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.483377 22153 tablet_bootstrap.cc:654] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.494308 22153 tablet_bootstrap.cc:492] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:18.494715 22153 ts_tablet_manager.cc:1403] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20251023 02:04:18.494639 21858 catalog_manager.cc:5649] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 2 to 3, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 3 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:18.497189 22153 raft_consensus.cc:359] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.497778 22153 raft_consensus.cc:385] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.498030 22153 raft_consensus.cc:740] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.498624 22153 consensus_queue.cc:260] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.500949 22153 ts_tablet_manager.cc:1434] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251023 02:04:18.501765 22153 tablet_bootstrap.cc:492] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:18.504271 22151 ts_tablet_manager.cc:1434] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.027s user 0.002s sys 0.003s
I20251023 02:04:18.507242 22153 tablet_bootstrap.cc:654] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.508136 22151 tablet_bootstrap.cc:492] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:18.514540 22151 tablet_bootstrap.cc:654] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.523765 22152 tablet_bootstrap.cc:492] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:18.523576 22316 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:18.524559 22152 ts_tablet_manager.cc:1403] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.047s user 0.020s sys 0.007s
I20251023 02:04:18.527880 22152 raft_consensus.cc:359] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.528896 22152 raft_consensus.cc:385] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.529289 22152 raft_consensus.cc:740] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.530491 22152 consensus_queue.cc:260] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.531728 22316 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.532245 22316 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.535799 22316 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.535712 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:18.536273 22151 tablet_bootstrap.cc:492] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:18.536701 22151 ts_tablet_manager.cc:1403] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.002s
I20251023 02:04:18.539345 22152 ts_tablet_manager.cc:1434] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
W20251023 02:04:18.540467 21928 leader_election.cc:343] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.542654 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:18.543105 22171 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:18.543275 22107 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:18.544575 21925 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:18.545724 22171 raft_consensus.cc:2804] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:18.546074 22171 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:18.546378 22171 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.539057 22151 raft_consensus.cc:359] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.551087 22151 raft_consensus.cc:385] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.551360 22151 raft_consensus.cc:740] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.551975 22151 consensus_queue.cc:260] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.554242 22151 ts_tablet_manager.cc:1434] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20251023 02:04:18.552845 22171 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.555099 22151 tablet_bootstrap.cc:492] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:18.555943 22171 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.556728 22189 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.557344 22189 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.557813 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:18.558440 22106 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.559824 22189 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.561470 22151 tablet_bootstrap.cc:654] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.561797 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.562376 21961 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:18.562388 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:18.563759 22152 tablet_bootstrap.cc:492] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:18.564133 22107 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:18.565510 22000 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:18.568739 22189 raft_consensus.cc:2804] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:18.569216 22189 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:18.572656 22189 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.573897 22151 tablet_bootstrap.cc:492] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:18.574339 22151 ts_tablet_manager.cc:1403] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20251023 02:04:18.575502 22034 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.569806 22152 tablet_bootstrap.cc:654] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.581552 22303 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.583037 22106 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:18.584823 22303 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.583371 22189 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.599524 22151 raft_consensus.cc:359] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.600190 22151 raft_consensus.cc:385] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.600435 22151 raft_consensus.cc:740] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.601442 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:18.597867 21925 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:18.598821 22153 tablet_bootstrap.cc:492] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:18.601065 22151 consensus_queue.cc:260] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.602634 22153 ts_tablet_manager.cc:1403] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.101s user 0.021s sys 0.024s
I20251023 02:04:18.590169 22303 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.603271 22036 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.604557 22151 ts_tablet_manager.cc:1434] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.030s user 0.012s sys 0.001s
I20251023 02:04:18.605912 22189 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.606771 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:18.607820 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.609990 22036 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:18.609962 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:18.606808 22202 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.610589 21961 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.610860 22202 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.611959 22153 raft_consensus.cc:359] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.612607 22153 raft_consensus.cc:385] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.612865 22153 raft_consensus.cc:740] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.613245 22202 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:18.613528 22153 consensus_queue.cc:260] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.614636 22106 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:18.615922 22153 ts_tablet_manager.cc:1434] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20251023 02:04:18.616802 22153 tablet_bootstrap.cc:492] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:18.617982 22316 raft_consensus.cc:2804] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:18.618455 22316 raft_consensus.cc:697] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:04:18.619617 22002 leader_election.cc:343] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.619351 22316 consensus_queue.cc:237] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.618373 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.623453 21959 raft_consensus.cc:2393] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 1: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:18.623764 22000 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:18.621229 22151 tablet_bootstrap.cc:492] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:18.625046 22107 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.628175 22303 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:18.628587 22303 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:18.628947 22303 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.632951 22151 tablet_bootstrap.cc:654] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.621030 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:18.633819 22106 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.636394 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:18.638530 22152 tablet_bootstrap.cc:492] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:18.638797 22153 tablet_bootstrap.cc:654] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.639006 22152 ts_tablet_manager.cc:1403] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.003s
I20251023 02:04:18.616770 21961 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:18.641424 22152 raft_consensus.cc:359] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.642259 22152 raft_consensus.cc:385] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.642603 22152 raft_consensus.cc:740] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.642747 22002 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:18.643603 22189 raft_consensus.cc:2804] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:18.643383 22152 consensus_queue.cc:260] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.644039 22189 raft_consensus.cc:697] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:18.645020 22189 consensus_queue.cc:237] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.646018 22152 ts_tablet_manager.cc:1434] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20251023 02:04:18.646621 22106 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:18.646835 22152 tablet_bootstrap.cc:492] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:18.647670 22158 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.648123 22158 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.650123 22158 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.651851 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:18.652436 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:18.652468 22109 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
W20251023 02:04:18.653568 21928 leader_election.cc:343] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.654462 22077 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:18.654649 21925 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:18.655306 22192 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.655686 22202 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:18.655730 22192 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.658245 22192 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.659449 22192 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:18.659775 22192 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:18.660135 22192 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.662793 22303 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.663534 22185 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.663561 22153 tablet_bootstrap.cc:492] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:18.672942 22153 ts_tablet_manager.cc:1403] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.056s user 0.016s sys 0.003s
I20251023 02:04:18.672797 22185 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.674813 22185 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.675256 22185 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.675637 22185 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.675915 22153 raft_consensus.cc:359] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.676142 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.667234 22152 tablet_bootstrap.cc:654] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.667868 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:18.677704 22034 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:18.666980 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:18.676525 22153 raft_consensus.cc:385] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.678366 22153 raft_consensus.cc:740] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.679131 22153 consensus_queue.cc:260] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:18.679551 21925 leader_election.cc:343] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.679432 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.670277 22345 consensus_queue.cc:1048] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:18.680220 21959 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:18.677839 22185 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.681423 22153 ts_tablet_manager.cc:1434] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20251023 02:04:18.682315 22153 tablet_bootstrap.cc:492] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:18.681744 22002 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:18.693488 21961 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:18.710352 22189 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.681536 21928 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:18.697420 22153 tablet_bootstrap.cc:654] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.698881 22152 tablet_bootstrap.cc:492] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:18.709049 22346 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:18.711089 22346 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:18.711226 22152 ts_tablet_manager.cc:1403] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.065s user 0.011s sys 0.005s
I20251023 02:04:18.711766 22158 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:18.712180 22158 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:18.712050 22189 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.711439 22346 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:18.702706 22077 consensus_peers.cc:597] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Couldn't send request to peer 818907d63f1f4d84ba267ebe6ee837e0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:18.703028 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:18.713403 22107 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.713797 22152 raft_consensus.cc:359] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.714463 22152 raft_consensus.cc:385] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.714728 22152 raft_consensus.cc:740] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.715112 22189 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.715335 22152 consensus_queue.cc:260] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.716984 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:18.700486 22303 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.704778 22202 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:18.705909 22151 tablet_bootstrap.cc:492] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:18.707010 21860 catalog_manager.cc:5649] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:18.681437 22185 consensus_queue.cc:1048] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20251023 02:04:18.710167 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:18.712611 22158 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.717640 22108 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:18.718096 22151 ts_tablet_manager.cc:1403] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.097s user 0.016s sys 0.027s
I20251023 02:04:18.726754 22151 raft_consensus.cc:359] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.727430 22151 raft_consensus.cc:385] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.727692 22151 raft_consensus.cc:740] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.728307 22151 consensus_queue.cc:260] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.700558 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:18.689422 22192 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.717556 22152 ts_tablet_manager.cc:1434] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251023 02:04:18.733925 22151 ts_tablet_manager.cc:1434] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251023 02:04:18.703814 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:18.736195 21858 catalog_manager.cc:5649] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:18.733880 22109 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:18.736256 22152 tablet_bootstrap.cc:492] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:18.742482 22158 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.747654 22151 tablet_bootstrap.cc:492] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:18.753059 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:18.753676 22034 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.753774 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:18.749454 22316 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:18.755167 21925 leader_election.cc:343] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.743412 22000 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:18.758244 22000 leader_election.cc:343] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.759464 22185 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:18.757529 22316 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.759960 22185 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:18.750039 22158 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.762614 22185 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.766033 22191 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:18.766677 22191 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.769910 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:18.770486 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:18.771376 21928 leader_election.cc:343] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
W20251023 02:04:18.771836 21925 leader_election.cc:343] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.772001 22185 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.772326 21925 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:18.773327 22158 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:18.774955 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:18.775511 22109 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.779467 22185 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.769280 22191 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.786553 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:18.787365 22036 raft_consensus.cc:2393] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 1: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:18.788281 22152 tablet_bootstrap.cc:654] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.791131 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:18.791770 21961 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:18.792200 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:18.792806 22107 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:18.793933 22000 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:18.794934 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:18.795527 22106 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.796321 22034 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:18.797197 22316 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.799746 21962 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:18.799872 22002 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:18.800762 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:18.801061 22002 leader_election.cc:343] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.801496 22345 raft_consensus.cc:2764] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251023 02:04:18.802322 21925 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:18.803304 21928 leader_election.cc:343] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:18.806376 22109 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:18.808488 22192 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.809386 21928 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:18.810024 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:18.810202 22192 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:18.810678 22192 raft_consensus.cc:697] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:18.810796 22346 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.811697 22106 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:18.811517 22192 consensus_queue.cc:237] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.812770 21925 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:18.812974 22346 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:18.813860 22191 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:18.814316 22191 raft_consensus.cc:697] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:18.814213 22000 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:18.815343 22303 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:18.815127 22191 consensus_queue.cc:237] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.815809 22303 raft_consensus.cc:697] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:18.816203 22151 tablet_bootstrap.cc:654] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:18.809757 22316 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.816591 22303 consensus_queue.cc:237] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.817797 22348 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:18.823324 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:18.821149 22348 raft_consensus.cc:697] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:18.827118 22106 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.827683 22189 consensus_queue.cc:1059] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20251023 02:04:18.860270 22153 tablet_bootstrap.cc:492] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:18.860872 22153 ts_tablet_manager.cc:1403] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.179s user 0.010s sys 0.008s
I20251023 02:04:18.882951 22151 tablet_bootstrap.cc:492] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:18.883576 22151 ts_tablet_manager.cc:1403] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.138s user 0.022s sys 0.014s
I20251023 02:04:18.904495 22345 raft_consensus.cc:3055] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:18.904973 22345 raft_consensus.cc:740] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:18.905872 22345 consensus_queue.cc:260] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.875140 22348 consensus_queue.cc:237] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.885367 22303 consensus_queue.cc:1048] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20251023 02:04:18.905498 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:18.908548 21962 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:18.914840 22316 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:18.929818 22303 consensus_queue.cc:1048] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:18.932813 22153 raft_consensus.cc:359] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.933595 22153 raft_consensus.cc:385] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.933933 22153 raft_consensus.cc:740] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.934587 22153 consensus_queue.cc:260] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.948498 21959 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:18.950906 21962 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:18.952778 22106 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:18.953707 22151 raft_consensus.cc:359] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.958429 22151 raft_consensus.cc:385] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.959451 22151 raft_consensus.cc:740] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.960152 22151 consensus_queue.cc:260] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.953683 22002 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:18.962019 22153 ts_tablet_manager.cc:1434] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.101s user 0.019s sys 0.006s
I20251023 02:04:18.962343 22185 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:18.962966 22185 raft_consensus.cc:697] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:18.963338 22152 tablet_bootstrap.cc:492] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:18.963770 22152 ts_tablet_manager.cc:1403] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.228s user 0.011s sys 0.016s
I20251023 02:04:18.964013 22185 consensus_queue.cc:237] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.965265 22345 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:18.965659 22153 tablet_bootstrap.cc:492] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:18.983637 22151 ts_tablet_manager.cc:1434] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.100s user 0.019s sys 0.013s
I20251023 02:04:18.984599 22151 tablet_bootstrap.cc:492] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:18.990034 22152 raft_consensus.cc:359] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.990923 22152 raft_consensus.cc:385] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:18.991223 22152 raft_consensus.cc:740] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:18.991770 22152 consensus_queue.cc:260] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:18.995441 22316 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.002688 22316 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:19.008018 21908 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251023 02:04:19.011078 21860 catalog_manager.cc:5649] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:19.014093 21860 catalog_manager.cc:5649] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:19.019567 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.021013 22034 raft_consensus.cc:1275] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
W20251023 02:04:19.021118 21928 leader_election.cc:343] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.021450 22034 pending_rounds.cc:85] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting all ops after (but not including) 1
I20251023 02:04:19.021791 22034 pending_rounds.cc:107] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251023 02:04:19.022082 22034 raft_consensus.cc:2889] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:04:19.022708 22152 ts_tablet_manager.cc:1434] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.059s user 0.006s sys 0.001s
I20251023 02:04:19.023589 22152 tablet_bootstrap.cc:492] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:19.024027 22153 tablet_bootstrap.cc:654] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.025463 22185 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.009446 22316 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.027801 22243 consensus_queue.cc:1048] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:19.056321 22110 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:19.030164 22185 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.076728 22364 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.081771 21962 raft_consensus.cc:1275] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:19.083397 22152 tablet_bootstrap.cc:654] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.077247 22364 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.079377 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:19.091398 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.092231 22364 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:19.091405 21925 leader_election.cc:343] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.093184 21925 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:19.094802 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:19.094879 22364 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:19.092950 22243 consensus_queue.cc:1048] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251023 02:04:19.095181 21928 leader_election.cc:343] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
W20251023 02:04:19.096596 21925 leader_election.cc:343] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.097076 21925 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:19.100581 22151 tablet_bootstrap.cc:654] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.106251 22153 tablet_bootstrap.cc:492] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:19.106367 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:19.106729 22153 ts_tablet_manager.cc:1403] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.144s user 0.010s sys 0.004s
W20251023 02:04:19.107731 22002 leader_election.cc:343] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.109169 22153 raft_consensus.cc:359] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.110015 22153 raft_consensus.cc:385] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.109647 22316 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:19.110308 22153 raft_consensus.cc:740] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.110934 22153 consensus_queue.cc:260] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.113525 22153 ts_tablet_manager.cc:1434] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251023 02:04:19.116720 22151 tablet_bootstrap.cc:492] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:19.117161 22151 ts_tablet_manager.cc:1403] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.133s user 0.004s sys 0.012s
I20251023 02:04:19.117192 22153 tablet_bootstrap.cc:492] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:19.125232 22364 consensus_queue.cc:1048] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.126065 22185 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.132268 22036 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.139770 21857 catalog_manager.cc:5649] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:19.140827 22152 tablet_bootstrap.cc:492] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:19.141301 22153 tablet_bootstrap.cc:654] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.141999 21857 catalog_manager.cc:5649] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:19.141304 22152 ts_tablet_manager.cc:1403] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.118s user 0.011s sys 0.002s
I20251023 02:04:19.146183 22152 raft_consensus.cc:359] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.146821 22152 raft_consensus.cc:385] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.147094 22152 raft_consensus.cc:740] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.147643 22152 consensus_queue.cc:260] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.149333 22152 ts_tablet_manager.cc:1434] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 02:04:19.150193 22152 tablet_bootstrap.cc:492] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:19.154055 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:19.154464 22151 raft_consensus.cc:359] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.155124 22151 raft_consensus.cc:385] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251023 02:04:19.155390 22000 leader_election.cc:343] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.155840 22000 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:19.156706 22359 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:19.155378 22151 raft_consensus.cc:740] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.157430 22316 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.157883 22316 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.158097 22364 consensus_queue.cc:1048] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.159713 22316 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.161124 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:19.161897 22106 raft_consensus.cc:2393] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 1: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:19.162739 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.163321 22151 consensus_queue.cc:260] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.165940 22151 ts_tablet_manager.cc:1434] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.044s user 0.002s sys 0.005s
I20251023 02:04:19.166636 22356 consensus_queue.cc:1048] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:19.167328 21928 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:19.167930 22153 tablet_bootstrap.cc:492] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:19.168089 22364 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.168421 22153 ts_tablet_manager.cc:1403] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.004s
I20251023 02:04:19.171152 22151 tablet_bootstrap.cc:492] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:19.174090 22153 raft_consensus.cc:359] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.174759 22153 raft_consensus.cc:385] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.175021 22153 raft_consensus.cc:740] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.175714 22153 consensus_queue.cc:260] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.178179 22356 consensus_queue.cc:1048] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.181437 22151 tablet_bootstrap.cc:654] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.184320 22152 tablet_bootstrap.cc:654] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.184460 22364 raft_consensus.cc:2749] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:19.188836 22153 ts_tablet_manager.cc:1434] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20251023 02:04:19.189986 22153 tablet_bootstrap.cc:492] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:19.192785 21858 catalog_manager.cc:5649] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:19.196981 22152 tablet_bootstrap.cc:492] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:19.197400 22152 ts_tablet_manager.cc:1403] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.007s
I20251023 02:04:19.199801 22152 raft_consensus.cc:359] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.200420 22152 raft_consensus.cc:385] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.200675 22152 raft_consensus.cc:740] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.203552 22153 tablet_bootstrap.cc:654] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.206152 22152 consensus_queue.cc:260] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.211177 22152 ts_tablet_manager.cc:1434] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.013s user 0.004s sys 0.001s
I20251023 02:04:19.212353 22152 tablet_bootstrap.cc:492] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:19.218598 22152 tablet_bootstrap.cc:654] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.224306 22151 tablet_bootstrap.cc:492] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:19.224766 22151 ts_tablet_manager.cc:1403] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.002s
I20251023 02:04:19.227762 22151 raft_consensus.cc:359] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.228395 22151 raft_consensus.cc:385] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.229542 22151 raft_consensus.cc:740] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.230258 22151 consensus_queue.cc:260] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.232594 22151 ts_tablet_manager.cc:1434] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20251023 02:04:19.239109 22364 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.239565 22364 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.241348 22364 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.242678 22151 tablet_bootstrap.cc:492] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:19.242832 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:19.244990 21925 leader_election.cc:343] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.246022 22152 tablet_bootstrap.cc:492] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:19.246356 22152 ts_tablet_manager.cc:1403] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.000s
I20251023 02:04:19.247555 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.248136 22036 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:19.248442 22152 raft_consensus.cc:359] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.249071 22152 raft_consensus.cc:385] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.249106 21928 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:19.249461 22152 raft_consensus.cc:740] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.250286 22364 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:19.250635 22364 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:19.250964 22364 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.254549 22185 consensus_queue.cc:1048] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.261135 22151 tablet_bootstrap.cc:654] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.250766 22152 consensus_queue.cc:260] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.248960 22153 tablet_bootstrap.cc:492] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:19.264492 22153 ts_tablet_manager.cc:1403] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.075s user 0.017s sys 0.005s
I20251023 02:04:19.263515 22364 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.266475 22152 ts_tablet_manager.cc:1434] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20251023 02:04:19.267982 22152 tablet_bootstrap.cc:492] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:19.269129 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:19.269816 22036 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.275063 22152 tablet_bootstrap.cc:654] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.277454 22191 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.277817 22034 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.277987 22191 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.279997 22191 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.281922 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.282614 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:19.283685 21928 leader_election.cc:343] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
W20251023 02:04:19.284411 21925 leader_election.cc:343] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.284960 21925 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:19.286309 22191 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:19.291726 22364 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.292305 22152 tablet_bootstrap.cc:492] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:19.292626 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:19.294721 22152 ts_tablet_manager.cc:1403] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.006s
W20251023 02:04:19.295246 21925 leader_election.cc:343] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.295145 22153 raft_consensus.cc:359] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.296097 22153 raft_consensus.cc:385] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.297053 22153 raft_consensus.cc:740] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.293048 22191 consensus_queue.cc:1048] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.293885 22036 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:19.299497 21928 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:19.299687 22152 raft_consensus.cc:359] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.309372 22152 raft_consensus.cc:385] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.309629 22152 raft_consensus.cc:740] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.309168 22153 consensus_queue.cc:260] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.311697 22153 ts_tablet_manager.cc:1434] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.047s user 0.005s sys 0.000s
I20251023 02:04:19.312597 22191 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:19.312638 22359 consensus_queue.cc:1048] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.313133 22191 raft_consensus.cc:697] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:19.312806 22152 consensus_queue.cc:260] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.313947 22191 consensus_queue.cc:237] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.315553 22152 ts_tablet_manager.cc:1434] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20251023 02:04:19.316483 22152 tablet_bootstrap.cc:492] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:19.322949 22153 tablet_bootstrap.cc:492] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:19.334750 22151 tablet_bootstrap.cc:492] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:19.335175 22151 ts_tablet_manager.cc:1403] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.093s user 0.010s sys 0.017s
I20251023 02:04:19.336285 21857 catalog_manager.cc:5649] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:19.342895 22151 raft_consensus.cc:359] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.343447 22151 raft_consensus.cc:385] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.343693 22151 raft_consensus.cc:740] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.344247 22151 consensus_queue.cc:260] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.348340 22151 ts_tablet_manager.cc:1434] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20251023 02:04:19.349969 22151 tablet_bootstrap.cc:492] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:19.350236 22152 tablet_bootstrap.cc:654] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.350603 22356 consensus_queue.cc:1048] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.361124 22153 tablet_bootstrap.cc:654] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.365619 22316 consensus_queue.cc:1048] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.365938 22243 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.366369 22243 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.367425 22202 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.367888 22202 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.368256 22243 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:19.369673 22202 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:19.369787 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:19.370366 21961 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:19.370357 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.371436 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.371645 22076 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:19.372160 22036 raft_consensus.cc:2393] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 1: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:04:19.372233 22077 leader_election.cc:343] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.372581 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:19.374042 22076 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:19.375394 22202 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:19.375819 22202 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:19.376219 22202 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.376669 22243 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.380375 22151 tablet_bootstrap.cc:654] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.384131 22243 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:19.387876 22202 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.390529 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:19.391072 21961 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.393930 22316 consensus_queue.cc:1048] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.398553 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:19.399912 21961 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:19.400235 22243 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:19.400251 22077 leader_election.cc:343] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.400681 22243 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.401028 22076 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:19.401857 22305 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:19.402359 22305 raft_consensus.cc:697] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:19.402698 22243 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:19.403148 22305 consensus_queue.cc:237] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.403970 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.404655 22036 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:19.405771 22077 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:19.406996 22210 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:19.407274 22210 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:19.407522 22210 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.411767 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:19.412501 21962 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:19.413856 22316 consensus_queue.cc:1048] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.418551 22153 tablet_bootstrap.cc:492] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:19.419080 22153 ts_tablet_manager.cc:1403] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.096s user 0.010s sys 0.006s
I20251023 02:04:19.419004 22202 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:19.419803 22210 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.421809 22153 raft_consensus.cc:359] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.422437 22153 raft_consensus.cc:385] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.422724 22153 raft_consensus.cc:740] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.423333 22153 consensus_queue.cc:260] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.425109 22151 tablet_bootstrap.cc:492] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:19.425482 22153 ts_tablet_manager.cc:1434] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20251023 02:04:19.425535 22151 ts_tablet_manager.cc:1403] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.076s user 0.009s sys 0.003s
I20251023 02:04:19.426934 22152 tablet_bootstrap.cc:492] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:19.431488 22153 tablet_bootstrap.cc:492] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:19.431770 22356 consensus_queue.cc:1048] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.432073 22152 ts_tablet_manager.cc:1403] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.116s user 0.012s sys 0.000s
I20251023 02:04:19.434440 22151 raft_consensus.cc:359] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.435129 22151 raft_consensus.cc:385] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.435387 22151 raft_consensus.cc:740] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.436024 22151 consensus_queue.cc:260] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.437355 22153 tablet_bootstrap.cc:654] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.438277 22151 ts_tablet_manager.cc:1434] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20251023 02:04:19.439229 22151 tablet_bootstrap.cc:492] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:19.444424 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:19.445047 22034 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.446926 22151 tablet_bootstrap.cc:654] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.447340 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:19.448019 21961 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.449996 22153 tablet_bootstrap.cc:492] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:19.450539 22153 ts_tablet_manager.cc:1403] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20251023 02:04:19.451148 22152 raft_consensus.cc:359] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.455479 22152 raft_consensus.cc:385] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.456707 22152 raft_consensus.cc:740] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.462716 22153 raft_consensus.cc:359] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.463601 22153 raft_consensus.cc:385] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.463944 22153 raft_consensus.cc:740] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.466948 22359 consensus_queue.cc:1048] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.475172 22359 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.479837 22158 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.480481 22158 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.491806 22210 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:19.495951 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.496151 22034 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
W20251023 02:04:19.497243 21928 leader_election.cc:343] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.497311 22077 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:19.498343 22210 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:19.498904 22158 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.499037 22210 raft_consensus.cc:697] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:19.499114 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:19.500000 22153 consensus_queue.cc:260] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:19.500514 21925 leader_election.cc:343] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.500970 21925 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:19.475714 22359 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.501832 22153 ts_tablet_manager.cc:1434] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.050s user 0.011s sys 0.004s
I20251023 02:04:19.502194 22158 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:19.503243 22153 tablet_bootstrap.cc:492] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:19.504598 22151 tablet_bootstrap.cc:492] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:19.504347 22189 consensus_queue.cc:1048] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:19.505033 22151 ts_tablet_manager.cc:1403] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.066s user 0.023s sys 0.022s
I20251023 02:04:19.502900 22152 consensus_queue.cc:260] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.507899 22152 ts_tablet_manager.cc:1434] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.073s user 0.002s sys 0.008s
I20251023 02:04:19.508666 22151 raft_consensus.cc:359] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.513825 22151 raft_consensus.cc:385] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.514133 22151 raft_consensus.cc:740] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.514371 22152 tablet_bootstrap.cc:492] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:19.499986 22210 consensus_queue.cc:237] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.509439 22153 tablet_bootstrap.cc:654] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.514856 22151 consensus_queue.cc:260] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.519431 22151 ts_tablet_manager.cc:1434] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.014s user 0.003s sys 0.006s
I20251023 02:04:19.520853 22151 tablet_bootstrap.cc:492] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:19.523290 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:19.523955 21959 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:19.524350 22359 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.524284 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:19.524842 22002 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:04:19.525898 22000 leader_election.cc:343] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.527802 22345 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:19.528318 22345 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:19.528653 22345 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.533164 22191 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.533670 22191 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.536576 22191 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.544687 22345 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.545389 22202 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.545852 22202 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.546553 22345 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.545253 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.561028 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.561678 22034 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:19.562480 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:19.569983 21928 leader_election.cc:343] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.571795 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:19.572705 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:04:19.615779 22141 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.20.81.62:41229, user_credentials={real_user=slave}} blocked reactor thread for 46547.6us
I20251023 02:04:19.623489 21959 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
W20251023 02:04:19.615397 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 46315.5us
I20251023 02:04:19.588527 21860 catalog_manager.cc:5649] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:19.615478 21961 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:19.553231 22202 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:19.625370 22000 leader_election.cc:343] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.572592 22077 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:19.626870 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 54405.9us
W20251023 02:04:19.572264 21925 leader_election.cc:343] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.628939 21925 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:19.629734 22191 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:19.630285 22202 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:19.630671 22152 tablet_bootstrap.cc:654] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.630789 22202 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:19.631160 22202 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:19.632493 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 60296.9us
I20251023 02:04:19.593734 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:19.634229 21960 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.613267 22151 tablet_bootstrap.cc:654] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.645673 21960 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:19.664662 22002 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:19.665542 22359 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:19.669176 22359 raft_consensus.cc:697] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:19.670353 22359 consensus_queue.cc:237] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.688340 22153 tablet_bootstrap.cc:492] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:19.688817 22153 ts_tablet_manager.cc:1403] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.186s user 0.022s sys 0.004s
I20251023 02:04:19.690657 22152 tablet_bootstrap.cc:492] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:19.691094 22152 ts_tablet_manager.cc:1403] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.177s user 0.027s sys 0.019s
I20251023 02:04:19.691532 22210 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.692016 22210 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.694852 22153 raft_consensus.cc:359] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.695515 22153 raft_consensus.cc:385] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.695791 22153 raft_consensus.cc:740] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.696445 22153 consensus_queue.cc:260] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.699045 22151 tablet_bootstrap.cc:492] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:19.699465 22151 ts_tablet_manager.cc:1403] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.179s user 0.010s sys 0.012s
I20251023 02:04:19.700103 22202 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.701754 22151 raft_consensus.cc:359] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.702394 22151 raft_consensus.cc:385] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.702658 22151 raft_consensus.cc:740] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.704761 22153 ts_tablet_manager.cc:1434] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20251023 02:04:19.705650 22153 tablet_bootstrap.cc:492] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:19.706084 22210 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:19.811173 22202 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:19.710089 22151 consensus_queue.cc:260] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:19.864167 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 66657us
I20251023 02:04:19.724589 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:19.786311 22345 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.863715 22152 raft_consensus.cc:359] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.867370 22151 ts_tablet_manager.cc:1434] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.168s user 0.046s sys 0.062s
I20251023 02:04:19.868564 22152 raft_consensus.cc:385] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:19.743850 21860 catalog_manager.cc:5649] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:19.868904 22152 raft_consensus.cc:740] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:19.869764 22151 tablet_bootstrap.cc:492] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:19.869719 22152 consensus_queue.cc:260] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.868906 22153 tablet_bootstrap.cc:654] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:19.813138 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:19.712020 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.865793 22171 consensus_queue.cc:1048] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251023 02:04:19.713047 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:19.872386 22243 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.879743 22243 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.881520 22243 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:19.874313 22033 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.889262 22033 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:19.890610 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:19.879001 22077 leader_election.cc:343] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.871954 21960 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.871910 22345 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.923319 21858 catalog_manager.cc:5649] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
W20251023 02:04:19.918466 22076 leader_election.cc:343] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.927326 22356 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:19.946578 22151 tablet_bootstrap.cc:654] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
W20251023 02:04:19.948047 22076 leader_election.cc:343] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:19.935920 22356 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:19.949740 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:19.950603 22036 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:19.954638 22152 ts_tablet_manager.cc:1434] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.263s user 0.010s sys 0.030s
I20251023 02:04:19.955703 22152 tablet_bootstrap.cc:492] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:19.953747 22076 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:19.959026 22305 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:19.993870 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:19.994609 22111 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:19.996874 22345 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.996984 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:19.999460 21956 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:19.998112 22077 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:20.002012 22403 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:20.002080 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:20.002673 22403 raft_consensus.cc:697] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:19.981352 22356 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:19.998849 22000 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:20.024008 21960 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:20.020542 22356 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:20.029422 22356 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:20.029810 22356 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.005309 22403 consensus_queue.cc:237] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:20.009863 22077 leader_election.cc:343] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.036800 22077 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:20.037545 22305 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:20.041697 22243 consensus_queue.cc:1048] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:20.070293 22210 consensus_queue.cc:1048] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:20.075050 22152 tablet_bootstrap.cc:654] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.065831 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:20.098413 21960 raft_consensus.cc:2393] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 1: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:20.103439 22151 tablet_bootstrap.cc:492] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:20.103821 22151 ts_tablet_manager.cc:1403] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.234s user 0.019s sys 0.010s
I20251023 02:04:20.104694 22153 tablet_bootstrap.cc:492] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:20.105031 22153 ts_tablet_manager.cc:1403] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.400s user 0.005s sys 0.025s
W20251023 02:04:20.100958 21925 consensus_peers.cc:597] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 -> Peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Couldn't send request to peer ca1362b9d9b34073bf35b6f6c9d1159a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:20.103134 22002 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:20.108312 22345 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:20.112962 22356 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.113722 21858 catalog_manager.cc:5649] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:20.115200 22356 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.122809 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:20.123142 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:20.123394 22108 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.123641 21960 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.126555 22151 raft_consensus.cc:359] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.127213 22151 raft_consensus.cc:385] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.127506 22151 raft_consensus.cc:740] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.128079 22151 consensus_queue.cc:260] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.121660 22210 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20251023 02:04:20.139686 22151 ts_tablet_manager.cc:1434] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20251023 02:04:20.140383 22151 tablet_bootstrap.cc:492] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:20.134622 21960 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:20.145687 22305 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:20.145700 22002 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:20.156461 22356 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:20.158824 22356 raft_consensus.cc:697] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:20.160398 22153 raft_consensus.cc:359] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.162142 22153 raft_consensus.cc:385] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.162456 22153 raft_consensus.cc:740] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.155253 22108 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:20.160423 22356 consensus_queue.cc:237] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.166771 22153 consensus_queue.cc:260] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.169900 22153 ts_tablet_manager.cc:1434] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.065s user 0.006s sys 0.001s
I20251023 02:04:20.173383 22153 tablet_bootstrap.cc:492] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:20.177861 22152 tablet_bootstrap.cc:492] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:20.178300 22152 ts_tablet_manager.cc:1403] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.223s user 0.011s sys 0.012s
I20251023 02:04:20.181042 22152 raft_consensus.cc:359] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.181733 22152 raft_consensus.cc:385] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.182013 22152 raft_consensus.cc:740] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.182650 22152 consensus_queue.cc:260] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.184934 22152 ts_tablet_manager.cc:1434] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251023 02:04:20.185833 22152 tablet_bootstrap.cc:492] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:20.188349 22151 tablet_bootstrap.cc:654] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
W20251023 02:04:20.192768 22413 log.cc:927] Time spent T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20251023 02:04:20.209942 22408 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.210474 22408 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.218894 22153 tablet_bootstrap.cc:654] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.238368 22408 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.245776 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:20.252878 21857 catalog_manager.cc:5649] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:20.250612 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:20.258256 22109 raft_consensus.cc:2468] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
W20251023 02:04:20.254791 22002 leader_election.cc:343] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.269120 22407 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.269609 22407 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.272330 22407 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.273319 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:20.270202 22000 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:04:20.274780 21928 leader_election.cc:343] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.274853 22408 raft_consensus.cc:2804] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:20.275472 22408 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:20.275828 22408 raft_consensus.cc:3060] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:20.275923 22000 consensus_peers.cc:597] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 -> Peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Couldn't send request to peer ca1362b9d9b34073bf35b6f6c9d1159a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:20.278626 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:20.280050 21925 leader_election.cc:343] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.280493 21925 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:20.281138 22189 consensus_queue.cc:1048] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:20.281519 22407 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:20.284615 22152 tablet_bootstrap.cc:654] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.299605 22151 tablet_bootstrap.cc:492] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:20.300081 22151 ts_tablet_manager.cc:1403] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.160s user 0.009s sys 0.016s
I20251023 02:04:20.310897 22151 raft_consensus.cc:359] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.313711 22151 raft_consensus.cc:385] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.314013 22151 raft_consensus.cc:740] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.314579 22151 consensus_queue.cc:260] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.322037 22151 ts_tablet_manager.cc:1434] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20251023 02:04:20.324361 22153 tablet_bootstrap.cc:492] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:20.324806 22153 ts_tablet_manager.cc:1403] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.152s user 0.014s sys 0.000s
I20251023 02:04:20.327557 22153 raft_consensus.cc:359] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.328155 22153 raft_consensus.cc:385] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.328393 22153 raft_consensus.cc:740] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.328955 22153 consensus_queue.cc:260] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.333107 22151 tablet_bootstrap.cc:492] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
W20251023 02:04:20.333109 21908 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251023 02:04:20.321936 22408 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.341430 22408 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.343221 22153 ts_tablet_manager.cc:1434] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.018s user 0.006s sys 0.002s
I20251023 02:04:20.344385 22153 tablet_bootstrap.cc:492] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:20.346405 22152 tablet_bootstrap.cc:492] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:20.346865 22152 ts_tablet_manager.cc:1403] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.161s user 0.006s sys 0.014s
I20251023 02:04:20.345963 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:20.347522 22110 raft_consensus.cc:3060] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.345127 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:20.351920 22002 leader_election.cc:343] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.356492 22152 raft_consensus.cc:359] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.357154 22152 raft_consensus.cc:385] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.357424 22152 raft_consensus.cc:740] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.358141 22151 tablet_bootstrap.cc:654] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.358321 22152 consensus_queue.cc:260] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.360342 22110 raft_consensus.cc:2468] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:20.361390 22000 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:20.362504 22356 raft_consensus.cc:2804] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:20.363003 22356 raft_consensus.cc:697] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:20.363778 22151 tablet_bootstrap.cc:492] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:20.363741 22356 consensus_queue.cc:237] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.364358 22151 ts_tablet_manager.cc:1403] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.000s
I20251023 02:04:20.369969 22151 raft_consensus.cc:359] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.370479 22151 raft_consensus.cc:385] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.370702 22151 raft_consensus.cc:740] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.370069 22153 tablet_bootstrap.cc:654] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.371232 22151 consensus_queue.cc:260] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.373260 22151 ts_tablet_manager.cc:1434] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251023 02:04:20.374112 22151 tablet_bootstrap.cc:492] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:20.374150 22403 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.374678 22403 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.375361 22152 ts_tablet_manager.cc:1434] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20251023 02:04:20.376252 22152 tablet_bootstrap.cc:492] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:20.376534 22403 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:20.378176 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:20.378764 22202 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.379191 22202 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:20.379823 22076 leader_election.cc:343] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.380586 22151 tablet_bootstrap.cc:654] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.380234 21860 catalog_manager.cc:5649] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:20.381002 22202 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:20.382045 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:20.382660 21957 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:20.384694 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:20.384917 22076 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:20.385866 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:20.386181 22202 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:20.386809 22202 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:04:20.387293 22077 leader_election.cc:343] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.387315 22202 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:20.388275 22077 leader_election.cc:343] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.388707 22077 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:20.389667 22243 raft_consensus.cc:2749] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:20.391640 22153 tablet_bootstrap.cc:492] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:20.392066 22153 ts_tablet_manager.cc:1403] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.000s
I20251023 02:04:20.394374 22153 raft_consensus.cc:359] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.394999 22153 raft_consensus.cc:385] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.395262 22153 raft_consensus.cc:740] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.397536 22202 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.401271 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:20.402007 21957 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.408221 22152 tablet_bootstrap.cc:654] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.410182 22153 consensus_queue.cc:260] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.412629 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:20.413928 22077 leader_election.cc:343] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.415633 21957 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:20.416749 22076 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:20.413975 22153 ts_tablet_manager.cc:1434] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20251023 02:04:20.417544 22403 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:20.418205 22403 raft_consensus.cc:697] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:20.418602 22153 tablet_bootstrap.cc:492] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:20.419296 22403 consensus_queue.cc:237] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.420885 22405 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.421346 22405 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.422986 22405 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.423799 22202 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:20.425649 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:20.426095 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:20.426306 22110 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:20.426903 22033 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:20.428310 21928 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:20.429167 22405 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:20.429466 22405 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:20.429773 22405 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.430838 22153 tablet_bootstrap.cc:654] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.435859 22152 tablet_bootstrap.cc:492] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:20.436281 22152 ts_tablet_manager.cc:1403] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.019s
I20251023 02:04:20.439203 22189 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.439538 22152 raft_consensus.cc:359] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.439695 22189 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.440240 22152 raft_consensus.cc:385] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.440531 22152 raft_consensus.cc:740] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.441126 22152 consensus_queue.cc:260] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.441630 22189 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.442821 22405 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.443640 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:20.443995 22153 tablet_bootstrap.cc:492] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:20.444249 21959 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:20.444429 22153 ts_tablet_manager.cc:1403] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.002s
I20251023 02:04:20.445544 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:20.445976 22002 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:20.446206 22110 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.446533 22405 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.447017 22153 raft_consensus.cc:359] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.447958 22345 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:20.448122 22153 raft_consensus.cc:385] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.448338 22345 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:20.448426 22153 raft_consensus.cc:740] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.448751 22345 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.451014 22152 ts_tablet_manager.cc:1434] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20251023 02:04:20.451915 22152 tablet_bootstrap.cc:492] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:20.453106 22110 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:20.449076 22153 consensus_queue.cc:260] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.454010 21925 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:20.454681 22405 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:20.455034 22405 raft_consensus.cc:697] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:20.455260 22153 ts_tablet_manager.cc:1434] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20251023 02:04:20.455729 22405 consensus_queue.cc:237] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.456115 22153 tablet_bootstrap.cc:492] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:20.460196 22152 tablet_bootstrap.cc:654] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.461288 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:20.447335 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:20.463465 22033 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.463685 22153 tablet_bootstrap.cc:654] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.468385 22153 tablet_bootstrap.cc:492] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:20.468782 22153 ts_tablet_manager.cc:1403] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20251023 02:04:20.470500 22153 raft_consensus.cc:359] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.470991 22153 raft_consensus.cc:385] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.471187 22153 raft_consensus.cc:740] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.471697 22153 consensus_queue.cc:260] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:20.473543 22000 leader_election.cc:343] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.478570 22153 ts_tablet_manager.cc:1434] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251023 02:04:20.479447 22153 tablet_bootstrap.cc:492] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:20.479537 22345 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.483431 22151 tablet_bootstrap.cc:492] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:20.484036 22151 ts_tablet_manager.cc:1403] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.110s user 0.017s sys 0.023s
I20251023 02:04:20.486163 22033 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:20.488336 22151 raft_consensus.cc:359] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.489212 22151 raft_consensus.cc:385] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.489501 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:20.490100 21957 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.489449 22151 raft_consensus.cc:740] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.491763 22406 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.496469 22151 consensus_queue.cc:260] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.495195 21860 catalog_manager.cc:5649] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:20.496932 22406 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.504417 22153 tablet_bootstrap.cc:654] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.511232 22406 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.511623 21857 catalog_manager.cc:5649] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:20.513754 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:20.518806 22151 ts_tablet_manager.cc:1434] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20251023 02:04:20.515966 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:20.519697 22151 tablet_bootstrap.cc:492] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:20.519942 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:20.520196 22000 leader_election.cc:343] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.520676 22110 raft_consensus.cc:2393] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 1: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:20.522221 21925 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:20.502528 22345 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.525154 21957 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:20.525719 22406 raft_consensus.cc:3060] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.526639 22202 consensus_queue.cc:1048] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:20.528813 22403 consensus_queue.cc:1048] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:20.529866 22110 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.533077 22002 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:20.533876 22408 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:20.534288 22408 raft_consensus.cc:697] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:20.535586 22153 tablet_bootstrap.cc:492] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:20.535995 22153 ts_tablet_manager.cc:1403] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.000s
I20251023 02:04:20.535075 22408 consensus_queue.cc:237] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.540984 22152 tablet_bootstrap.cc:492] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:20.541544 22152 ts_tablet_manager.cc:1403] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.090s user 0.021s sys 0.033s
I20251023 02:04:20.549432 22153 raft_consensus.cc:359] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.550155 22153 raft_consensus.cc:385] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.550405 22153 raft_consensus.cc:740] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.551918 22153 consensus_queue.cc:260] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.570276 22151 tablet_bootstrap.cc:654] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.570816 22153 ts_tablet_manager.cc:1434] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.035s user 0.005s sys 0.001s
I20251023 02:04:20.571004 22152 raft_consensus.cc:359] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.571630 22152 raft_consensus.cc:385] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.571748 22153 tablet_bootstrap.cc:492] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:20.571905 22152 raft_consensus.cc:740] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.572450 22152 consensus_queue.cc:260] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.587996 22406 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:20.591967 22152 ts_tablet_manager.cc:1434] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.050s user 0.001s sys 0.005s
I20251023 02:04:20.592880 22152 tablet_bootstrap.cc:492] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:20.624464 22243 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:20.625108 22243 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.630820 22243 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:20.624049 22152 tablet_bootstrap.cc:654] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.634322 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:20.634322 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:20.637558 22076 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:20.638620 22202 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:20.630857 22153 tablet_bootstrap.cc:654] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.643421 21857 catalog_manager.cc:5649] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:20.646667 22345 consensus_queue.cc:1048] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:20.653101 22152 tablet_bootstrap.cc:492] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:20.654635 22152 ts_tablet_manager.cc:1403] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.062s user 0.012s sys 0.001s
I20251023 02:04:20.656728 22152 raft_consensus.cc:359] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.657291 22152 raft_consensus.cc:385] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.657545 22152 raft_consensus.cc:740] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.658149 22152 consensus_queue.cc:260] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.660305 22152 ts_tablet_manager.cc:1434] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251023 02:04:20.661105 22152 tablet_bootstrap.cc:492] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:20.663292 22151 tablet_bootstrap.cc:492] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:20.663725 22151 ts_tablet_manager.cc:1403] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.144s user 0.011s sys 0.009s
I20251023 02:04:20.665865 22406 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.666076 22151 raft_consensus.cc:359] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.666343 22406 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.666262 22202 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.667266 22151 raft_consensus.cc:385] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.668072 22151 raft_consensus.cc:740] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.668799 22151 consensus_queue.cc:260] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.670957 22406 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.667629 22202 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.673516 22202 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:20.662887 22408 consensus_queue.cc:1048] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:20.670934 22151 ts_tablet_manager.cc:1434] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20251023 02:04:20.674813 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:20.675365 22151 tablet_bootstrap.cc:492] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:20.675907 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:20.677129 21925 leader_election.cc:343] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
W20251023 02:04:20.677837 22077 leader_election.cc:343] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.678143 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:20.678937 22403 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.679380 22403 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:20.679674 22076 leader_election.cc:343] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.680115 22076 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:20.681545 22152 tablet_bootstrap.cc:654] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.681986 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:20.685783 22243 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:20.685999 21958 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:20.681303 22403 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:20.688587 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:20.688872 22076 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:20.689332 22034 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:20.690822 22202 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:20.691221 22202 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:20.691684 22202 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.693992 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:20.694662 22033 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:20.695784 21928 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:20.697716 22406 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:20.698148 22406 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:20.698590 22406 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.700846 22202 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.704121 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:20.704882 22033 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.706550 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:20.707288 21959 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.708895 22202 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:20.716743 22151 tablet_bootstrap.cc:654] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.720407 22153 tablet_bootstrap.cc:492] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:20.720863 22153 ts_tablet_manager.cc:1403] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.149s user 0.024s sys 0.001s
I20251023 02:04:20.721529 22202 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.722155 22202 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.724241 22153 raft_consensus.cc:359] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.724924 22153 raft_consensus.cc:385] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.725169 22153 raft_consensus.cc:740] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.727819 21959 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:20.727734 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:20.728924 22153 consensus_queue.cc:260] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.730335 22076 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:20.731001 22405 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.731446 22305 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:20.731405 22406 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.731459 22405 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.733255 22305 raft_consensus.cc:697] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:20.734315 22405 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.734355 22305 consensus_queue.cc:237] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.736191 22202 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:20.735553 22033 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:20.735088 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:20.738689 21928 leader_election.cc:343] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.736399 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:20.735486 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:20.740101 22153 ts_tablet_manager.cc:1434] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.019s user 0.006s sys 0.002s
W20251023 02:04:20.740329 21925 leader_election.cc:343] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.740869 22153 tablet_bootstrap.cc:492] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:20.740731 21925 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:20.741915 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:20.742461 22034 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.745999 22076 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:20.746961 22202 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:20.750073 22405 raft_consensus.cc:2749] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:20.750562 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:20.751083 22406 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:20.751736 21925 leader_election.cc:343] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:20.756268 22153 tablet_bootstrap.cc:654] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.758281 22034 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:20.759881 22152 tablet_bootstrap.cc:492] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:20.759577 21928 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:20.760368 22152 ts_tablet_manager.cc:1403] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.099s user 0.008s sys 0.013s
I20251023 02:04:20.761160 22151 tablet_bootstrap.cc:492] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:20.761693 22151 ts_tablet_manager.cc:1403] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.086s user 0.008s sys 0.004s
I20251023 02:04:20.761756 22406 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:20.762272 22406 raft_consensus.cc:697] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:20.762954 22152 raft_consensus.cc:359] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.763411 22152 raft_consensus.cc:385] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.763587 22152 raft_consensus.cc:740] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.763067 22406 consensus_queue.cc:237] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.764038 22151 raft_consensus.cc:359] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.764719 22151 raft_consensus.cc:385] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.764982 22151 raft_consensus.cc:740] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.765554 22151 consensus_queue.cc:260] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.766062 22153 tablet_bootstrap.cc:492] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:20.765712 22152 consensus_queue.cc:260] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.766646 22153 ts_tablet_manager.cc:1403] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.003s
I20251023 02:04:20.769234 22153 raft_consensus.cc:359] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.769902 22153 raft_consensus.cc:385] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.770198 22153 raft_consensus.cc:740] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.770817 22153 consensus_queue.cc:260] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.772923 22153 ts_tablet_manager.cc:1434] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251023 02:04:20.774048 22153 tablet_bootstrap.cc:492] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:20.779121 22151 ts_tablet_manager.cc:1434] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.017s user 0.001s sys 0.003s
I20251023 02:04:20.779620 22153 tablet_bootstrap.cc:654] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.780020 22151 tablet_bootstrap.cc:492] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:20.787292 22151 tablet_bootstrap.cc:654] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.791510 21858 catalog_manager.cc:5649] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:20.803364 22152 ts_tablet_manager.cc:1434] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251023 02:04:20.804340 22152 tablet_bootstrap.cc:492] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:20.806278 22108 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:20.833206 22153 tablet_bootstrap.cc:492] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:20.833971 22153 ts_tablet_manager.cc:1403] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.060s user 0.026s sys 0.027s
I20251023 02:04:20.835289 22356 consensus_queue.cc:1048] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:20.839371 22152 tablet_bootstrap.cc:654] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.840497 22153 raft_consensus.cc:359] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.841339 22153 raft_consensus.cc:385] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.841684 22153 raft_consensus.cc:740] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.845369 22153 consensus_queue.cc:260] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.848009 22153 ts_tablet_manager.cc:1434] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20251023 02:04:20.850996 22151 tablet_bootstrap.cc:492] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:20.851444 22151 ts_tablet_manager.cc:1403] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.072s user 0.011s sys 0.007s
I20251023 02:04:20.852573 22408 consensus_queue.cc:1048] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:20.854290 22153 tablet_bootstrap.cc:492] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:20.860010 22153 tablet_bootstrap.cc:654] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.846151 21857 catalog_manager.cc:5649] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:20.868144 22153 tablet_bootstrap.cc:492] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:20.868541 22153 ts_tablet_manager.cc:1403] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20251023 02:04:20.869262 22151 raft_consensus.cc:359] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.869918 22151 raft_consensus.cc:385] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.870167 22151 raft_consensus.cc:740] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.870326 22153 raft_consensus.cc:359] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.870913 22153 raft_consensus.cc:385] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.871168 22153 raft_consensus.cc:740] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.870816 22151 consensus_queue.cc:260] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.872005 22153 consensus_queue.cc:260] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.873716 22153 ts_tablet_manager.cc:1434] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20251023 02:04:20.874533 22153 tablet_bootstrap.cc:492] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:20.880410 22153 tablet_bootstrap.cc:654] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:20.930073 22153 tablet_bootstrap.cc:492] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:20.930591 22153 ts_tablet_manager.cc:1403] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.009s
I20251023 02:04:20.931890 22151 ts_tablet_manager.cc:1434] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.078s user 0.008s sys 0.004s
I20251023 02:04:20.933621 22151 tablet_bootstrap.cc:492] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:20.933835 22152 tablet_bootstrap.cc:492] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:20.934230 22152 ts_tablet_manager.cc:1403] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.130s user 0.008s sys 0.006s
I20251023 02:04:20.934846 22153 raft_consensus.cc:359] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.935595 22153 raft_consensus.cc:385] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.936025 22153 raft_consensus.cc:740] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.936755 22153 consensus_queue.cc:260] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.939589 22153 ts_tablet_manager.cc:1434] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251023 02:04:20.940465 22153 tablet_bootstrap.cc:492] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:20.952045 22407 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:20.952549 22407 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.955559 22407 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:20.957131 22153 tablet_bootstrap.cc:654] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
W20251023 02:04:20.962085 22077 consensus_peers.cc:597] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Couldn't send request to peer 818907d63f1f4d84ba267ebe6ee837e0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:20.963821 22152 raft_consensus.cc:359] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.964537 22152 raft_consensus.cc:385] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:20.964823 22152 raft_consensus.cc:740] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:20.965549 22152 consensus_queue.cc:260] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:20.967521 22202 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:20.976251 22152 ts_tablet_manager.cc:1434] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.040s user 0.002s sys 0.004s
I20251023 02:04:20.977476 22152 tablet_bootstrap.cc:492] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:20.986369 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:20.987025 22110 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:20.997973 21925 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:21.001809 22158 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:21.003803 22158 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:21.004142 22158 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.007154 22151 tablet_bootstrap.cc:654] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:21.002808 22406 consensus_queue.cc:1048] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:21.009852 22405 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:21.010867 22405 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.012830 22345 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:21.013262 22405 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:21.013290 22345 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.015839 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:21.016176 22345 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:21.016413 22034 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:21.013924 22158 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.018654 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:21.018874 22153 tablet_bootstrap.cc:492] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:21.019313 22153 ts_tablet_manager.cc:1403] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.079s user 0.014s sys 0.001s
I20251023 02:04:21.022418 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:21.022982 22108 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:21.023237 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:21.023947 21958 raft_consensus.cc:2393] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 1: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:21.024025 22000 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:21.024904 22461 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:21.024952 22158 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:21.025800 22152 tablet_bootstrap.cc:654] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:21.028492 22110 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.029840 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:21.025341 22461 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:21.032121 22461 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:21.034559 21925 leader_election.cc:343] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.034742 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:21.035290 22111 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.050593 22153 raft_consensus.cc:359] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.051962 22153 raft_consensus.cc:385] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:21.040486 22158 consensus_queue.cc:1048] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
W20251023 02:04:21.053678 21928 leader_election.cc:343] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.058259 22153 raft_consensus.cc:740] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:21.060554 22153 consensus_queue.cc:260] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.061725 21928 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:21.062716 22153 ts_tablet_manager.cc:1434] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.043s user 0.009s sys 0.000s
I20251023 02:04:21.062649 22356 consensus_queue.cc:1048] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:21.063642 22153 tablet_bootstrap.cc:492] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:21.064165 22158 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:21.068050 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:21.069777 22111 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:21.077154 21925 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:21.078627 22158 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:21.079521 22461 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.082034 22158 raft_consensus.cc:697] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:21.086355 22036 raft_consensus.cc:2393] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 1: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:21.089808 22151 tablet_bootstrap.cc:492] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:21.090327 22151 ts_tablet_manager.cc:1403] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.157s user 0.017s sys 0.008s
I20251023 02:04:21.095503 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:21.096282 22108 raft_consensus.cc:2393] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 1: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:21.096145 22158 consensus_queue.cc:237] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.098004 22461 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:21.101152 22345 consensus_queue.cc:1048] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:21.101233 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:21.113865 22151 raft_consensus.cc:359] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.114566 22151 raft_consensus.cc:385] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:21.114897 22151 raft_consensus.cc:740] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:21.115609 22151 consensus_queue.cc:260] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.117856 22151 ts_tablet_manager.cc:1434] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20251023 02:04:21.118696 22151 tablet_bootstrap.cc:492] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:21.125550 22002 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:21.126336 22461 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 02:04:21.129817 22152 tablet_bootstrap.cc:492] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:21.130270 22152 ts_tablet_manager.cc:1403] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.153s user 0.013s sys 0.007s
I20251023 02:04:21.152468 22151 tablet_bootstrap.cc:654] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:21.154295 22152 raft_consensus.cc:359] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.155388 22152 raft_consensus.cc:385] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:21.155587 22152 raft_consensus.cc:740] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:21.153843 22403 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:21.157953 22152 consensus_queue.cc:260] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.162493 22153 tablet_bootstrap.cc:654] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:21.158664 21860 catalog_manager.cc:5649] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:21.157738 22403 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.168623 22403 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:21.166970 22111 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.167013 22305 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251023 02:04:21.172782 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:21.174496 22076 leader_election.cc:343] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.182026 22407 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:21.185564 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:21.186334 22036 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:21.187811 22077 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:21.194344 22403 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:21.194700 22403 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:21.195020 22403 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.198911 22152 ts_tablet_manager.cc:1434] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.068s user 0.006s sys 0.001s
I20251023 02:04:21.200142 22152 tablet_bootstrap.cc:492] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:21.225462 22305 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:21.225978 22305 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.227805 22305 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:21.229278 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:21.230163 22036 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:21.231289 22077 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:21.215754 22403 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.232084 22152 tablet_bootstrap.cc:654] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:21.232740 22202 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:21.233098 22202 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:21.233399 22202 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.240264 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:21.238137 22405 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:21.240870 21959 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:21.240916 22405 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.242462 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:21.242898 22405 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:21.243182 22035 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.242053 22202 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.246246 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:21.247391 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:21.248039 22202 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:21.250269 22076 leader_election.cc:343] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.236151 22403 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:21.261164 21928 leader_election.cc:343] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.258332 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:21.270522 22151 tablet_bootstrap.cc:492] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:21.271040 22151 ts_tablet_manager.cc:1403] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.153s user 0.009s sys 0.007s
I20251023 02:04:21.275408 22469 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:21.275586 22210 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:21.278708 22035 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:21.270587 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:21.281670 22111 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:21.272725 21959 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.288820 22077 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:21.290625 22403 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:21.291085 22403 raft_consensus.cc:697] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:21.292032 22403 consensus_queue.cc:237] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.293718 22151 raft_consensus.cc:359] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.294497 22151 raft_consensus.cc:385] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:21.294902 22151 raft_consensus.cc:740] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:21.295744 22151 consensus_queue.cc:260] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.297304 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:21.298518 22033 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.298619 22469 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:21.299110 22469 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.300927 22469 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:21.301623 22153 tablet_bootstrap.cc:492] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:21.302028 22153 ts_tablet_manager.cc:1403] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.239s user 0.012s sys 0.027s
I20251023 02:04:21.303865 21925 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:21.307099 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:21.308041 22158 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:21.310458 22158 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:21.311086 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:21.311700 22110 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
W20251023 02:04:21.312880 21928 leader_election.cc:343] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.314409 21959 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:21.310967 22158 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.319977 22033 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:21.321017 22077 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:21.313611 21925 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:21.322041 22151 ts_tablet_manager.cc:1434] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.051s user 0.004s sys 0.003s
I20251023 02:04:21.322425 22469 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:21.322924 22469 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:21.323568 22469 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.322152 22202 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:21.324563 22202 raft_consensus.cc:697] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:21.325716 22202 consensus_queue.cc:237] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.327818 22153 raft_consensus.cc:359] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.328490 22153 raft_consensus.cc:385] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:21.328763 22153 raft_consensus.cc:740] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:21.329331 22153 consensus_queue.cc:260] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.331506 22152 tablet_bootstrap.cc:492] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:21.332062 22152 ts_tablet_manager.cc:1403] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.132s user 0.015s sys 0.023s
I20251023 02:04:21.332563 22151 tablet_bootstrap.cc:492] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:21.331517 22153 ts_tablet_manager.cc:1434] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20251023 02:04:21.338876 22152 raft_consensus.cc:359] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.342476 22152 raft_consensus.cc:385] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:21.342725 22152 raft_consensus.cc:740] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:21.343595 22152 consensus_queue.cc:260] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.331861 22158 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.345705 22152 ts_tablet_manager.cc:1434] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20251023 02:04:21.346601 22152 tablet_bootstrap.cc:492] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:21.353243 22153 tablet_bootstrap.cc:492] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:21.386904 22151 tablet_bootstrap.cc:654] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:21.392704 22158 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:21.395396 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:21.395969 22110 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.402297 22110 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:21.429320 22152 tablet_bootstrap.cc:654] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:21.405930 21860 catalog_manager.cc:5649] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:21.406450 21925 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:21.486198 22158 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:21.425486 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:21.406631 22408 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:21.487553 22158 raft_consensus.cc:697] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:04:21.489645 21928 leader_election.cc:343] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.489464 22158 consensus_queue.cc:237] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.490646 22153 tablet_bootstrap.cc:654] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
W20251023 02:04:21.486756 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 80415.1us
I20251023 02:04:21.429388 22469 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.446135 22403 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:21.495013 22403 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.497889 22403 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:21.501346 22151 tablet_bootstrap.cc:492] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:21.502031 22151 ts_tablet_manager.cc:1403] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.170s user 0.017s sys 0.005s
I20251023 02:04:21.498898 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:21.504518 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:21.505568 22077 leader_election.cc:343] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
W20251023 02:04:21.505941 21928 leader_election.cc:343] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.509089 22469 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:21.493081 22408 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.510725 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:21.510869 22033 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.511301 22110 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.521987 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:21.567760 22408 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:21.571522 22110 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:21.573278 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:21.573999 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:21.592733 22405 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:21.577114 22151 raft_consensus.cc:359] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.593221 22405 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.593715 22151 raft_consensus.cc:385] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:21.593999 22151 raft_consensus.cc:740] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:21.594601 22151 consensus_queue.cc:260] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.595158 22405 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:21.602676 21925 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:21.605370 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:21.606648 21860 catalog_manager.cc:5649] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
W20251023 02:04:21.607432 22076 leader_election.cc:343] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.607879 22076 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:21.608430 21908 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251023 02:04:21.609149 22403 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:21.610910 22406 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:21.611369 22406 raft_consensus.cc:697] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:21.612089 22406 consensus_queue.cc:237] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.607447 22002 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:21.620611 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:21.672466 22036 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
W20251023 02:04:21.712982 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 55513.7us
I20251023 02:04:21.725363 22151 ts_tablet_manager.cc:1434] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.223s user 0.021s sys 0.019s
I20251023 02:04:21.729079 22151 tablet_bootstrap.cc:492] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:21.730917 22189 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:21.735388 22305 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:21.735877 22152 tablet_bootstrap.cc:492] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:21.736812 22152 ts_tablet_manager.cc:1403] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.390s user 0.025s sys 0.036s
I20251023 02:04:21.736343 22305 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.749369 22305 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:21.755074 21928 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:21.758442 22487 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:21.759039 22487 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:21.760785 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:21.760566 22487 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:21.763439 22077 leader_election.cc:343] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
W20251023 02:04:21.764338 21925 leader_election.cc:343] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.767952 22153 tablet_bootstrap.cc:492] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:21.780627 22153 ts_tablet_manager.cc:1403] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.442s user 0.027s sys 0.059s
I20251023 02:04:21.774283 22151 tablet_bootstrap.cc:654] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:21.769783 22152 raft_consensus.cc:359] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.790657 22152 raft_consensus.cc:385] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:21.790987 22152 raft_consensus.cc:740] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:21.782766 22469 consensus_queue.cc:1048] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:21.793035 22487 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.794795 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:21.796355 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:21.796972 22035 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:21.778470 21860 catalog_manager.cc:5649] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:21.791688 22152 consensus_queue.cc:260] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.802317 22152 ts_tablet_manager.cc:1434] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.065s user 0.009s sys 0.002s
I20251023 02:04:21.787199 22153 raft_consensus.cc:359] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.802939 22153 raft_consensus.cc:385] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:21.803349 22153 raft_consensus.cc:740] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:21.803740 22152 tablet_bootstrap.cc:492] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:21.804292 22153 consensus_queue.cc:260] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:21.809788 22076 leader_election.cc:343] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.810314 22076 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:21.811581 22403 raft_consensus.cc:2749] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 02:04:21.817477 22076 consensus_peers.cc:597] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Couldn't send request to peer 02c53ba06cc3445a8534a84aa76b1c14. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:21.817600 22035 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:21.820948 22469 consensus_queue.cc:1048] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251023 02:04:21.829270 22153 ts_tablet_manager.cc:1434] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20251023 02:04:21.829696 21928 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:21.833767 22469 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:21.834251 22469 raft_consensus.cc:697] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:21.834112 22202 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:21.835189 22469 consensus_queue.cc:237] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.825534 22487 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:21.844064 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:04:21.851706 21925 leader_election.cc:343] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:21.865259 22151 tablet_bootstrap.cc:492] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:21.865993 22151 ts_tablet_manager.cc:1403] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.138s user 0.024s sys 0.014s
I20251023 02:04:21.866990 22153 tablet_bootstrap.cc:492] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:21.874248 22151 raft_consensus.cc:359] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.875259 22151 raft_consensus.cc:385] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:21.875576 22151 raft_consensus.cc:740] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:21.876231 22151 consensus_queue.cc:260] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.879688 22151 ts_tablet_manager.cc:1434] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20251023 02:04:21.881189 22151 tablet_bootstrap.cc:492] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:21.888854 22403 consensus_queue.cc:1048] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:21.898016 21857 catalog_manager.cc:5649] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:21.898489 22305 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251023 02:04:21.903259 22152 tablet_bootstrap.cc:654] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
W20251023 02:04:21.916597 21928 consensus_peers.cc:597] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 -> Peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Couldn't send request to peer 818907d63f1f4d84ba267ebe6ee837e0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:21.933048 22153 tablet_bootstrap.cc:654] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:21.934906 22151 tablet_bootstrap.cc:654] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:21.980137 22152 tablet_bootstrap.cc:492] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:21.980581 22152 ts_tablet_manager.cc:1403] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.177s user 0.013s sys 0.011s
I20251023 02:04:21.984550 22152 raft_consensus.cc:359] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:21.985124 22152 raft_consensus.cc:385] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:21.985392 22152 raft_consensus.cc:740] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:21.986184 22152 consensus_queue.cc:260] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.001642 22189 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.002079 22189 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.004474 22189 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.004964 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:22.005676 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:22.006500 22111 raft_consensus.cc:2393] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 1: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:22.005551 22151 tablet_bootstrap.cc:492] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:22.007141 22151 ts_tablet_manager.cc:1403] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.126s user 0.009s sys 0.004s
I20251023 02:04:22.007594 22000 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:22.009495 22189 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.011277 22151 raft_consensus.cc:359] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.011914 22151 raft_consensus.cc:385] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.012118 22151 raft_consensus.cc:740] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.012621 22151 consensus_queue.cc:260] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.013957 22152 ts_tablet_manager.cc:1434] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.033s user 0.009s sys 0.000s
I20251023 02:04:22.015013 22152 tablet_bootstrap.cc:492] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:22.017329 22153 tablet_bootstrap.cc:492] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:22.017784 22153 ts_tablet_manager.cc:1403] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.151s user 0.010s sys 0.004s
I20251023 02:04:22.018693 21857 catalog_manager.cc:5649] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:22.020934 22153 raft_consensus.cc:359] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.021554 22153 raft_consensus.cc:385] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.021829 22153 raft_consensus.cc:740] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.022480 22153 consensus_queue.cc:260] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.024585 22153 ts_tablet_manager.cc:1434] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20251023 02:04:22.026098 22151 ts_tablet_manager.cc:1434] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20251023 02:04:22.027127 22151 tablet_bootstrap.cc:492] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:22.027657 22403 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.028090 22403 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.031828 22403 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:22.031000 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:22.032609 21958 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:22.033834 22153 tablet_bootstrap.cc:492] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:22.035640 22076 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:22.036144 22189 raft_consensus.cc:2749] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:22.036957 22210 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:22.037165 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:22.037366 22210 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:22.037719 22152 tablet_bootstrap.cc:654] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.037806 22210 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.032917 22151 tablet_bootstrap.cc:654] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.042474 22153 tablet_bootstrap.cc:654] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
W20251023 02:04:22.042582 22077 leader_election.cc:343] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.052037 22210 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.052917 22151 tablet_bootstrap.cc:492] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:22.053355 22151 ts_tablet_manager.cc:1403] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.005s
I20251023 02:04:22.057297 22152 tablet_bootstrap.cc:492] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:22.057760 22152 ts_tablet_manager.cc:1403] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.007s
I20251023 02:04:22.064286 22152 raft_consensus.cc:359] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.064873 22151 raft_consensus.cc:359] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.065135 22152 raft_consensus.cc:385] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.065462 22151 raft_consensus.cc:385] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.065541 22152 raft_consensus.cc:740] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.076220 22152 consensus_queue.cc:260] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.075275 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:22.079353 21959 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.075397 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:22.080930 22077 leader_election.cc:343] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.075462 22210 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:22.088632 21959 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:22.089972 22076 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:22.091228 22210 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:22.091383 22153 tablet_bootstrap.cc:492] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:22.091866 22210 raft_consensus.cc:697] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:22.065766 22151 raft_consensus.cc:740] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.092736 22151 consensus_queue.cc:260] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.092876 22210 consensus_queue.cc:237] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.091847 22153 ts_tablet_manager.cc:1403] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.024s
I20251023 02:04:22.095381 22151 ts_tablet_manager.cc:1434] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.042s user 0.007s sys 0.000s
I20251023 02:04:22.097085 22153 raft_consensus.cc:359] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.097791 22153 raft_consensus.cc:385] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.098075 22153 raft_consensus.cc:740] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.098644 22153 consensus_queue.cc:260] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.098693 22151 tablet_bootstrap.cc:492] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:22.102464 22158 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:22.101292 22153 ts_tablet_manager.cc:1434] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20251023 02:04:22.104041 22153 tablet_bootstrap.cc:492] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:22.109762 22487 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.110234 22487 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.112149 22487 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.114826 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:22.115763 22152 ts_tablet_manager.cc:1434] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.058s user 0.004s sys 0.005s
I20251023 02:04:22.116954 22152 tablet_bootstrap.cc:492] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:22.120968 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:22.123733 22152 tablet_bootstrap.cc:654] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.124785 22151 tablet_bootstrap.cc:654] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.127950 22153 tablet_bootstrap.cc:654] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.129133 21928 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:22.130263 22487 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:22.152927 22487 consensus_queue.cc:1048] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:22.166932 22151 tablet_bootstrap.cc:492] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:22.167951 22151 ts_tablet_manager.cc:1403] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.069s user 0.014s sys 0.011s
I20251023 02:04:22.170869 22151 raft_consensus.cc:359] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.171495 22151 raft_consensus.cc:385] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.171761 22151 raft_consensus.cc:740] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.172328 22151 consensus_queue.cc:260] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.173156 21860 catalog_manager.cc:5649] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:22.175581 22151 ts_tablet_manager.cc:1434] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251023 02:04:22.175741 22469 consensus_queue.cc:1048] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:22.180521 22153 tablet_bootstrap.cc:492] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:22.180995 22153 ts_tablet_manager.cc:1403] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.077s user 0.011s sys 0.013s
I20251023 02:04:22.183586 22153 raft_consensus.cc:359] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.184226 22153 raft_consensus.cc:385] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.184509 22153 raft_consensus.cc:740] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.185518 22153 consensus_queue.cc:260] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.186105 22151 tablet_bootstrap.cc:492] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:22.192220 22151 tablet_bootstrap.cc:654] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.195734 22153 ts_tablet_manager.cc:1434] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20251023 02:04:22.199136 22152 tablet_bootstrap.cc:492] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:22.199611 22152 ts_tablet_manager.cc:1403] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.083s user 0.014s sys 0.024s
I20251023 02:04:22.202113 22152 raft_consensus.cc:359] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.202783 22152 raft_consensus.cc:385] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.203023 22152 raft_consensus.cc:740] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.196961 22153 tablet_bootstrap.cc:492] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:22.203660 22152 consensus_queue.cc:260] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.205910 22152 ts_tablet_manager.cc:1434] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20251023 02:04:22.207387 22152 tablet_bootstrap.cc:492] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:22.209875 22153 tablet_bootstrap.cc:654] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.216084 22152 tablet_bootstrap.cc:654] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.218111 22151 tablet_bootstrap.cc:492] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:22.218552 22151 ts_tablet_manager.cc:1403] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.011s
I20251023 02:04:22.220906 22151 raft_consensus.cc:359] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.221520 22151 raft_consensus.cc:385] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.221810 22151 raft_consensus.cc:740] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.222354 22151 consensus_queue.cc:260] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.224766 22151 ts_tablet_manager.cc:1434] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251023 02:04:22.225764 22151 tablet_bootstrap.cc:492] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:22.234498 22151 tablet_bootstrap.cc:654] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.236958 22153 tablet_bootstrap.cc:492] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:22.237401 22153 ts_tablet_manager.cc:1403] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.003s
I20251023 02:04:22.239892 22153 raft_consensus.cc:359] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.240556 22153 raft_consensus.cc:385] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.240811 22153 raft_consensus.cc:740] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.241387 22153 consensus_queue.cc:260] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.243574 22153 ts_tablet_manager.cc:1434] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20251023 02:04:22.244498 22153 tablet_bootstrap.cc:492] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:22.252300 22153 tablet_bootstrap.cc:654] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.252684 22305 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.253165 22305 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.255067 22305 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:22.256520 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:22.257097 22036 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:22.258816 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:22.260507 22077 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:22.261807 22076 leader_election.cc:343] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.262665 22305 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:22.263026 22305 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:22.263365 22305 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.271975 22461 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.272446 22461 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.275293 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:22.275925 22107 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:22.277563 22000 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:22.278299 22189 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:22.278622 22189 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:22.278903 22189 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.279356 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:22.280664 22002 leader_election.cc:343] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.281148 22461 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.270869 22305 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.284741 22151 tablet_bootstrap.cc:492] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:22.284797 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:22.285490 22151 ts_tablet_manager.cc:1403] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.005s
W20251023 02:04:22.286367 22076 leader_election.cc:343] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.287235 22189 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.288161 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:22.288268 22151 raft_consensus.cc:359] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.288591 22305 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:22.293933 22151 raft_consensus.cc:385] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.293928 22036 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.294595 22151 raft_consensus.cc:740] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.295646 22151 consensus_queue.cc:260] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.297791 22189 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.297957 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:22.298658 22151 ts_tablet_manager.cc:1434] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.013s user 0.005s sys 0.002s
I20251023 02:04:22.299515 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:04:22.299901 22002 leader_election.cc:343] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.300082 22107 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.303239 22151 tablet_bootstrap.cc:492] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:22.304167 22036 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:22.306824 22077 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:22.308130 22152 tablet_bootstrap.cc:492] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:22.308557 22210 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:22.309000 22210 raft_consensus.cc:697] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:22.310370 22153 tablet_bootstrap.cc:492] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:22.308558 22152 ts_tablet_manager.cc:1403] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.101s user 0.017s sys 0.025s
I20251023 02:04:22.311698 22151 tablet_bootstrap.cc:654] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.313581 22152 raft_consensus.cc:359] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.314284 22152 raft_consensus.cc:385] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.314643 22153 ts_tablet_manager.cc:1403] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.070s user 0.026s sys 0.017s
I20251023 02:04:22.314541 22210 consensus_queue.cc:237] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.318082 22153 raft_consensus.cc:359] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.319518 22153 raft_consensus.cc:385] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.319840 22153 raft_consensus.cc:740] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.314577 22152 raft_consensus.cc:740] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.321770 21959 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.320953 22152 consensus_queue.cc:260] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.326220 22107 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:22.327395 22000 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:22.328473 22461 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:22.328961 22461 raft_consensus.cc:697] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:22.329942 22461 consensus_queue.cc:237] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.332443 22152 ts_tablet_manager.cc:1434] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.021s user 0.001s sys 0.005s
I20251023 02:04:22.333356 22152 tablet_bootstrap.cc:492] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:22.335983 22189 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.336530 22189 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.338172 22189 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.339706 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:22.340368 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:22.340508 22111 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:22.341766 22000 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:22.341420 22153 consensus_queue.cc:260] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:22.342571 22002 leader_election.cc:343] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.342562 22345 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:22.343767 22345 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:22.344159 22345 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.344337 22153 ts_tablet_manager.cc:1434] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20251023 02:04:22.345539 22153 tablet_bootstrap.cc:492] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:22.352649 22153 tablet_bootstrap.cc:654] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.340653 22152 tablet_bootstrap.cc:654] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.354760 22111 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.365617 22345 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.370719 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:22.371295 22109 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.371601 22469 consensus_queue.cc:1048] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:22.385382 22109 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:22.387385 22000 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:22.390300 22524 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:22.399443 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:22.413686 22405 consensus_queue.cc:1048] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:22.414201 22461 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.414706 22461 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.417471 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:22.430500 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:22.431648 21958 raft_consensus.cc:2393] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 1: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:22.434481 22151 tablet_bootstrap.cc:492] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:22.435164 22151 ts_tablet_manager.cc:1403] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.132s user 0.036s sys 0.031s
I20251023 02:04:22.435889 22461 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.438591 22151 raft_consensus.cc:359] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.439656 22151 raft_consensus.cc:385] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.440095 22151 raft_consensus.cc:740] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.441084 22151 consensus_queue.cc:260] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.442831 22152 tablet_bootstrap.cc:492] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:22.443356 22152 ts_tablet_manager.cc:1403] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.110s user 0.015s sys 0.032s
I20251023 02:04:22.442948 21858 catalog_manager.cc:5649] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:22.444548 22151 ts_tablet_manager.cc:1434] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20251023 02:04:22.445652 22152 raft_consensus.cc:359] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.446329 22152 raft_consensus.cc:385] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.446610 22152 raft_consensus.cc:740] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.446035 22151 tablet_bootstrap.cc:492] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:22.447186 22152 consensus_queue.cc:260] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.449385 22152 ts_tablet_manager.cc:1434] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20251023 02:04:22.439016 21857 catalog_manager.cc:5649] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:22.456660 22152 tablet_bootstrap.cc:492] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:22.455438 22151 tablet_bootstrap.cc:654] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.468084 22151 tablet_bootstrap.cc:492] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:22.468796 22151 ts_tablet_manager.cc:1403] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.008s
I20251023 02:04:22.471676 22152 tablet_bootstrap.cc:654] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.471839 22151 raft_consensus.cc:359] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.473088 22151 raft_consensus.cc:385] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.473433 22151 raft_consensus.cc:740] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.474337 22151 consensus_queue.cc:260] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.477283 22151 ts_tablet_manager.cc:1434] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251023 02:04:22.479550 22151 tablet_bootstrap.cc:492] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:22.489495 22151 tablet_bootstrap.cc:654] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.491037 22469 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.491542 22469 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.494798 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:22.496165 21925 leader_election.cc:343] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.497740 22152 tablet_bootstrap.cc:492] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:22.498221 22152 ts_tablet_manager.cc:1403] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.003s
I20251023 02:04:22.500576 22152 raft_consensus.cc:359] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.501235 22152 raft_consensus.cc:385] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.502866 22210 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.493422 22469 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.503301 22210 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.505019 22210 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:22.507354 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:22.508788 22076 leader_election.cc:343] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.501495 22152 raft_consensus.cc:740] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.517343 22152 consensus_queue.cc:260] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:22.518152 22002 leader_election.cc:343] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.520474 22152 ts_tablet_manager.cc:1434] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20251023 02:04:22.521533 22152 tablet_bootstrap.cc:492] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:22.407577 22345 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.527263 22152 tablet_bootstrap.cc:654] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.531647 22151 tablet_bootstrap.cc:492] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:22.532387 22151 ts_tablet_manager.cc:1403] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.053s user 0.026s sys 0.012s
I20251023 02:04:22.537539 22524 raft_consensus.cc:697] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:22.540768 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:22.541656 22037 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:22.560552 21928 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:22.560711 22151 raft_consensus.cc:359] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.561614 22151 raft_consensus.cc:385] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.561990 22469 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:22.562587 22469 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:22.563050 22469 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.565711 22158 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.565451 22152 tablet_bootstrap.cc:492] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:22.566295 22158 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.566831 22152 ts_tablet_manager.cc:1403] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.006s
I20251023 02:04:22.569128 22158 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.561962 22151 raft_consensus.cc:740] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.570142 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:22.570281 22151 consensus_queue.cc:260] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.570725 22111 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:22.571507 21925 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:22.572232 22487 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:22.572558 22487 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:22.572912 22487 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.574163 22151 ts_tablet_manager.cc:1434] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.041s user 0.003s sys 0.004s
I20251023 02:04:22.575181 22151 tablet_bootstrap.cc:492] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:22.575789 22152 raft_consensus.cc:359] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.577448 22152 raft_consensus.cc:385] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.538846 22524 consensus_queue.cc:237] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.578233 22152 raft_consensus.cc:740] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.580271 22152 consensus_queue.cc:260] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.585299 22407 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.585758 22407 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.586675 22152 ts_tablet_manager.cc:1434] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20251023 02:04:22.586798 22487 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.587682 22407 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.588675 22153 tablet_bootstrap.cc:492] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:22.589169 22153 ts_tablet_manager.cc:1403] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.244s user 0.052s sys 0.063s
I20251023 02:04:22.589005 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:22.589695 22106 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:22.590752 21925 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:22.591524 22407 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:22.591886 22407 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:22.592751 22407 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.590248 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:22.595265 21928 leader_election.cc:343] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.596524 22487 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.594692 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:22.597730 22152 tablet_bootstrap.cc:492] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:22.597800 22153 raft_consensus.cc:359] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.665772 22153 raft_consensus.cc:385] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.666064 22153 raft_consensus.cc:740] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.623700 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:22.667699 22153 consensus_queue.cc:260] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.666504 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:22.617951 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:22.668128 22106 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
W20251023 02:04:22.680410 21928 leader_election.cc:343] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
W20251023 02:04:22.682101 21928 leader_election.cc:343] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.683706 22153 ts_tablet_manager.cc:1434] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.092s user 0.036s sys 0.036s
I20251023 02:04:22.685883 22030 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.686839 22153 tablet_bootstrap.cc:492] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:22.684345 22000 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:22.665290 22469 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.677651 22407 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:22.711320 22077 leader_election.cc:343] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.711760 22077 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:22.712776 22469 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.713049 22210 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:22.726850 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:04:22.731351 21925 leader_election.cc:343] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.733268 22536 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.733814 22536 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.729432 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:22.734642 22305 consensus_queue.cc:1048] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251023 02:04:22.740665 22151 tablet_bootstrap.cc:654] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.745978 22407 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.747851 22153 tablet_bootstrap.cc:654] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.748692 22106 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:22.749441 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:22.749955 22524 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 02:04:22.751564 21928 leader_election.cc:343] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.751616 21925 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:22.738184 22035 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.754029 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:22.754576 22109 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.755234 22537 consensus_queue.cc:1048] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:22.752765 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:22.756561 21928 leader_election.cc:343] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.755841 22487 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:22.750021 22536 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.747363 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:22.758802 21925 leader_election.cc:343] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.759202 21925 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:22.764482 22035 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
W20251023 02:04:22.764773 21908 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251023 02:04:22.757321 22487 raft_consensus.cc:697] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:22.769907 22487 consensus_queue.cc:237] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.772020 22152 tablet_bootstrap.cc:654] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.765636 22109 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:22.763911 22407 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:22.774559 21925 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:22.768523 21860 catalog_manager.cc:5649] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:22.777184 22407 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:22.780004 22524 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.786938 21928 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:22.780931 22407 raft_consensus.cc:697] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:22.788002 22405 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:22.788452 22405 raft_consensus.cc:697] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:22.780431 22524 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.778262 22536 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.791432 22528 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.791888 22528 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.793910 22524 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.788319 22407 consensus_queue.cc:237] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.796429 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:22.797696 22002 leader_election.cc:343] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.797876 22405 consensus_queue.cc:237] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.821815 22151 tablet_bootstrap.cc:492] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:22.822295 22151 ts_tablet_manager.cc:1403] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.247s user 0.023s sys 0.028s
I20251023 02:04:22.828639 22530 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.831223 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:22.830914 22530 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.831907 21958 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:22.833783 22002 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:22.834494 22345 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:22.834830 22345 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:22.835170 22345 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.835796 22530 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.816190 22536 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.843617 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:22.851012 22536 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.845139 22528 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.852577 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:22.853073 22076 consensus_peers.cc:597] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Couldn't send request to peer 02c53ba06cc3445a8534a84aa76b1c14. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:22.853201 22107 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 0.
I20251023 02:04:22.854408 21925 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:22.855122 22536 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:22.855460 22536 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:22.855398 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:22.855895 22536 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.857105 21860 catalog_manager.cc:5649] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
W20251023 02:04:22.858880 21928 leader_election.cc:343] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.858506 21860 catalog_manager.cc:5649] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:22.859906 21860 catalog_manager.cc:5649] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:22.864627 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:22.864723 22345 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.876981 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:22.878185 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:22.878933 22109 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
W20251023 02:04:22.880342 22000 leader_election.cc:343] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.880739 22000 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:22.881444 22530 raft_consensus.cc:2749] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:22.886377 22153 tablet_bootstrap.cc:492] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:22.886837 22153 ts_tablet_manager.cc:1403] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.200s user 0.013s sys 0.018s
I20251023 02:04:22.890197 22345 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.889839 22153 raft_consensus.cc:359] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.893797 22000 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:22.893949 22153 raft_consensus.cc:385] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.894359 22153 raft_consensus.cc:740] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.894976 22153 consensus_queue.cc:260] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.896013 22530 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:22.896533 22536 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.897161 22153 ts_tablet_manager.cc:1434] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251023 02:04:22.898141 22153 tablet_bootstrap.cc:492] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:22.902500 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:22.903082 22109 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.903816 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:22.905654 21928 leader_election.cc:343] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.911059 22305 consensus_queue.cc:1048] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:22.913136 22153 tablet_bootstrap.cc:654] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.916010 22109 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:22.919469 22000 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:22.919864 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:22.921077 22111 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.921232 22528 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:22.921768 22528 raft_consensus.cc:697] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:22.922533 22528 consensus_queue.cc:237] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.922740 22151 raft_consensus.cc:359] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.923342 22151 raft_consensus.cc:385] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.923609 22151 raft_consensus.cc:740] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.926085 22151 consensus_queue.cc:260] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.927987 22151 ts_tablet_manager.cc:1434] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.104s user 0.004s sys 0.000s
I20251023 02:04:22.930241 22111 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:22.931406 21925 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:22.932498 22405 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.933104 22469 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:22.932958 22405 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.933611 22469 raft_consensus.cc:697] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:22.934795 22405 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.936198 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:22.936581 21959 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.936717 21958 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.934444 22469 consensus_queue.cc:237] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.947325 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:22.947737 22151 tablet_bootstrap.cc:492] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:22.948102 22109 raft_consensus.cc:2393] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 1: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:22.951987 21958 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:22.955137 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:22.891772 22152 tablet_bootstrap.cc:492] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:22.956183 22403 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:22.956796 22152 ts_tablet_manager.cc:1403] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.359s user 0.026s sys 0.005s
I20251023 02:04:22.956652 22403 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.959178 22403 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:22.959260 21928 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:22.960276 22487 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:22.964459 22151 tablet_bootstrap.cc:654] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:22.953547 22345 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251023 02:04:22.973562 21857 catalog_manager.cc:5649] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:22.979712 21860 catalog_manager.cc:5649] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:22.968008 22153 tablet_bootstrap.cc:492] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:22.969573 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:22.982821 22153 ts_tablet_manager.cc:1403] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.085s user 0.017s sys 0.000s
I20251023 02:04:22.961371 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:22.981815 22528 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251023 02:04:22.984138 22076 leader_election.cc:343] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:22.966101 22536 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:22.984822 22035 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 0.
I20251023 02:04:22.986990 22077 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:04:22.987728 22002 consensus_peers.cc:597] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 -> Peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Couldn't send request to peer 02c53ba06cc3445a8534a84aa76b1c14. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:22.987958 22403 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:22.988220 22035 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.988345 22403 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:22.988903 22403 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:22.997224 22153 raft_consensus.cc:359] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.997943 22153 raft_consensus.cc:385] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:22.998204 22153 raft_consensus.cc:740] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:22.998792 22153 consensus_queue.cc:260] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:22.999096 22403 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.001037 22153 ts_tablet_manager.cc:1434] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20251023 02:04:23.002102 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:23.006049 22403 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:23.007004 22076 leader_election.cc:343] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Illegal state: must be running to vote when last-logged opid is not known
I20251023 02:04:23.007176 22153 tablet_bootstrap.cc:492] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a: Bootstrap starting.
I20251023 02:04:23.008412 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:23.009028 22030 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:23.014374 22152 raft_consensus.cc:359] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.016330 22152 raft_consensus.cc:385] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:23.016640 22152 raft_consensus.cc:740] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:23.017686 22153 tablet_bootstrap.cc:654] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:23.018608 22030 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:23.017340 22152 consensus_queue.cc:260] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.021275 22152 ts_tablet_manager.cc:1434] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.063s user 0.004s sys 0.002s
I20251023 02:04:23.023065 22077 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:23.028261 22151 tablet_bootstrap.cc:492] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:23.030709 22153 tablet_bootstrap.cc:492] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a: No bootstrap required, opened a new log
I20251023 02:04:23.032613 22151 ts_tablet_manager.cc:1403] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.101s user 0.025s sys 0.016s
I20251023 02:04:23.032670 22153 ts_tablet_manager.cc:1403] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.002s
I20251023 02:04:23.033046 22403 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:23.033686 22152 tablet_bootstrap.cc:492] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:23.034133 22403 raft_consensus.cc:697] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:23.034945 22403 consensus_queue.cc:237] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.035146 22151 raft_consensus.cc:359] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.035816 22151 raft_consensus.cc:385] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:23.036077 22151 raft_consensus.cc:740] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:23.036649 22151 consensus_queue.cc:260] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.039053 22151 ts_tablet_manager.cc:1434] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251023 02:04:23.039918 22151 tablet_bootstrap.cc:492] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:23.039995 22152 tablet_bootstrap.cc:654] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:23.047385 22153 raft_consensus.cc:359] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.048022 22153 raft_consensus.cc:385] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:23.048311 22153 raft_consensus.cc:740] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Initialized, Role: FOLLOWER
I20251023 02:04:23.048875 22153 consensus_queue.cc:260] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.051633 22153 ts_tablet_manager.cc:1434] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent starting tablet: real 0.018s user 0.001s sys 0.003s
I20251023 02:04:23.048162 22151 tablet_bootstrap.cc:654] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:23.063969 22152 tablet_bootstrap.cc:492] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:23.065519 22152 ts_tablet_manager.cc:1403] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.004s
I20251023 02:04:23.069192 22152 raft_consensus.cc:359] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.070089 22152 raft_consensus.cc:385] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:23.071520 22152 raft_consensus.cc:740] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:23.072311 22152 consensus_queue.cc:260] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.075181 22152 ts_tablet_manager.cc:1434] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251023 02:04:23.076385 22152 tablet_bootstrap.cc:492] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14: Bootstrap starting.
I20251023 02:04:23.079345 22151 tablet_bootstrap.cc:492] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:23.079823 22151 ts_tablet_manager.cc:1403] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.040s user 0.018s sys 0.006s
I20251023 02:04:23.081375 21860 catalog_manager.cc:5649] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:23.082551 22152 tablet_bootstrap.cc:654] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:23.082281 22151 raft_consensus.cc:359] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.082949 22151 raft_consensus.cc:385] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:23.083333 22151 raft_consensus.cc:740] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:23.083986 22151 consensus_queue.cc:260] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.101295 22151 ts_tablet_manager.cc:1434] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20251023 02:04:23.102283 22151 tablet_bootstrap.cc:492] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0: Bootstrap starting.
I20251023 02:04:23.107739 22151 tablet_bootstrap.cc:654] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0: Neither blocks nor log segments found. Creating new log.
I20251023 02:04:23.125550 22152 tablet_bootstrap.cc:492] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14: No bootstrap required, opened a new log
I20251023 02:04:23.127013 22152 ts_tablet_manager.cc:1403] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent bootstrapping tablet: real 0.051s user 0.017s sys 0.008s
I20251023 02:04:23.128115 22151 tablet_bootstrap.cc:492] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0: No bootstrap required, opened a new log
I20251023 02:04:23.128530 22151 ts_tablet_manager.cc:1403] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.015s
I20251023 02:04:23.129415 22152 raft_consensus.cc:359] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.130092 22152 raft_consensus.cc:385] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:23.130316 22152 raft_consensus.cc:740] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Initialized, Role: FOLLOWER
I20251023 02:04:23.130832 22151 raft_consensus.cc:359] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.130838 22152 consensus_queue.cc:260] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.131558 22151 raft_consensus.cc:385] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 02:04:23.131872 22151 raft_consensus.cc:740] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Initialized, Role: FOLLOWER
I20251023 02:04:23.132419 22151 consensus_queue.cc:260] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.134279 22151 ts_tablet_manager.cc:1434] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251023 02:04:23.134281 22152 ts_tablet_manager.cc:1434] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20251023 02:04:23.162597 22345 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:23.163126 22345 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.164683 22345 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:23.165409 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:23.165648 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:23.166344 22109 raft_consensus.cc:2393] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 1: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:23.167201 22000 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:23.167836 22345 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:23.171813 22345 raft_consensus.cc:2749] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:23.190475 22345 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:23.228251 22469 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:23.228668 22469 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.230366 22469 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:23.231122 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:23.231923 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:23.233675 21925 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:23.234227 22469 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 02:04:23.267313 22030 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:23.267459 22469 consensus_queue.cc:1048] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:23.274395 22469 consensus_queue.cc:1048] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:23.280702 22469 consensus_queue.cc:1048] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:23.288244 22487 consensus_queue.cc:1048] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:23.307402 22487 consensus_queue.cc:1048] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:23.311853 22523 consensus_queue.cc:1048] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 02:04:23.325165 22523 consensus_queue.cc:1048] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:23.325596 22107 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:23.336086 22469 consensus_queue.cc:1048] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:23.388513 22523 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:23.388923 22523 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.390499 22523 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:23.391132 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:23.391389 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:23.391769 21958 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:23.391980 22107 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 0.
I20251023 02:04:23.392704 22002 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:23.393429 22523 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 02:04:23.393851 22523 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:23.394244 22523 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:23.397189 22035 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:23.399926 22523 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.401304 22523 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:23.401988 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:23.402257 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:23.402621 21958 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:23.402766 22107 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:23.403555 22469 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:23.408870 21958 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:23.412499 22407 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:23.414649 21960 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:23.416358 22107 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:23.417733 22000 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:23.418509 22523 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 02:04:23.418939 22523 raft_consensus.cc:697] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:23.419699 22523 consensus_queue.cc:237] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:23.434478 21958 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:23.437219 21858 catalog_manager.cc:5649] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:23.456998 21959 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 0 FOLLOWER]: Advancing to term 1
I20251023 02:04:23.473675 22403 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:23.487268 22305 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:23.852376 22345 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:23.867152 22345 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 02:04:24.504896 20804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 02:04:24.547597 22579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:04:24.547636 22578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 02:04:24.549346 22581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 02:04:24.550541 20804 server_base.cc:1047] running on GCE node
I20251023 02:04:24.551873 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251023 02:04:24.552103 20804 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251023 02:04:24.552330 20804 hybrid_clock.cc:648] HybridClock initialized: now 1761185064552311 us; error 0 us; skew 500 ppm
I20251023 02:04:24.552913 20804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 02:04:24.556427 20804 webserver.cc:492] Webserver started at http://127.20.81.4:38509/ using document root <none> and password file <none>
I20251023 02:04:24.557086 20804 fs_manager.cc:362] Metadata directory not provided
I20251023 02:04:24.557407 20804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 02:04:24.557842 20804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 02:04:24.559463 20804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-3-root/instance:
uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
format_stamp: "Formatted at 2025-10-23 02:04:24 on dist-test-slave-6q7l"
I20251023 02:04:24.566617 20804 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20251023 02:04:24.569895 22586 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:24.570670 20804 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251023 02:04:24.570955 20804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-3-root
uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
format_stamp: "Formatted at 2025-10-23 02:04:24 on dist-test-slave-6q7l"
I20251023 02:04:24.571219 20804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 02:04:24.593357 20804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 02:04:24.595059 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 02:04:24.596933 20804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 02:04:24.600023 20804 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 02:04:24.600199 20804 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:24.600466 20804 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 02:04:24.600630 20804 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 02:04:24.653993 20804 rpc_server.cc:307] RPC server started. Bound to: 127.20.81.4:33255
I20251023 02:04:24.654109 22648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.81.4:33255 every 8 connection(s)
I20251023 02:04:24.657826 20804 auto_leader_rebalancer-test.cc:417] add a tserver: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:24.669001 22649 heartbeater.cc:344] Connected to a master server at 127.20.81.62:41229
I20251023 02:04:24.669457 22649 heartbeater.cc:461] Registering TS with master...
I20251023 02:04:24.670184 22649 heartbeater.cc:507] Master 127.20.81.62:41229 requested a full tablet report, sending...
I20251023 02:04:24.671876 21858 ts_manager.cc:194] Registered new tserver with Master: 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:24.673113 21858 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33304
I20251023 02:04:24.891360 21959 consensus_queue.cc:237] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:24.898317 22030 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:24.899094 22107 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:24.900238 22407 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:24.902180 22487 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:24.915896 22487 raft_consensus.cc:2955] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:24.917644 22030 raft_consensus.cc:2955] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
W20251023 02:04:24.935022 21926 consensus_peers.cc:597] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 972d07041fbd4097854ef23c73f09c75. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:24.941455 21958 consensus_queue.cc:237] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:24.945263 22107 raft_consensus.cc:2955] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:24.979383 22033 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251023 02:04:24.981462 21926 consensus_peers.cc:597] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 365d47bf062b4ebabb6d5f247e4e3e33. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:24.981710 22656 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:24.982745 22107 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:24.983939 22656 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:24.993472 21858 catalog_manager.cc:5649] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:24.998029 22661 raft_consensus.cc:2955] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:25.008217 22035 raft_consensus.cc:2955] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:25.025336 22107 raft_consensus.cc:2955] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:25.043838 22030 consensus_queue.cc:237] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:25.062316 21958 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:25.063655 22461 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:25.064059 22111 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:25.066057 22345 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:25.064284 21858 catalog_manager.cc:5649] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:25.086526 22461 raft_consensus.cc:2955] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:25.089366 21958 raft_consensus.cc:2955] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
W20251023 02:04:25.099759 22001 consensus_peers.cc:597] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 774c010489aa4775bb6d636926900529. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:25.108337 21860 catalog_manager.cc:5649] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:25.115605 22030 consensus_queue.cc:237] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
W20251023 02:04:25.124751 22001 consensus_peers.cc:597] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e32bb6c0622f44f8806f77eee0691856. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:25.127560 21958 raft_consensus.cc:1275] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:25.129805 22678 consensus_queue.cc:1048] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:25.130833 22109 raft_consensus.cc:1275] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:25.140836 22109 raft_consensus.cc:2955] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:25.148036 22345 raft_consensus.cc:2955] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:25.150660 21958 raft_consensus.cc:2955] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:25.156438 22345 consensus_queue.cc:1048] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:25.179435 22107 raft_consensus.cc:2955] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:25.181913 21858 catalog_manager.cc:5649] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:25.322149 22682 ts_tablet_manager.cc:933] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499)
I20251023 02:04:25.324884 22682 tablet_copy_client.cc:323] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.1:41499
I20251023 02:04:25.338145 21972 tablet_copy_service.cc:140] P 02c53ba06cc3445a8534a84aa76b1c14: Received BeginTabletCopySession request for tablet 972d07041fbd4097854ef23c73f09c75 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:36488)
I20251023 02:04:25.338757 21972 tablet_copy_service.cc:161] P 02c53ba06cc3445a8534a84aa76b1c14: Beginning new tablet copy session on tablet 972d07041fbd4097854ef23c73f09c75 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:36488: session id = 1ab8b0b58147446d84d901b4ed2ff39e-972d07041fbd4097854ef23c73f09c75
I20251023 02:04:25.344703 21972 tablet_copy_source_session.cc:215] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:25.348299 22682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 972d07041fbd4097854ef23c73f09c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:25.363535 22682 tablet_copy_client.cc:806] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:25.364190 22682 tablet_copy_client.cc:670] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:25.369022 22682 tablet_copy_client.cc:538] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:25.377208 22682 tablet_bootstrap.cc:492] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:25.400647 22686 ts_tablet_manager.cc:933] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499)
I20251023 02:04:25.405292 22686 tablet_copy_client.cc:323] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.1:41499
I20251023 02:04:25.405603 22682 tablet_bootstrap.cc:492] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:25.406464 22682 tablet_bootstrap.cc:492] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:25.407111 22682 ts_tablet_manager.cc:1403] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.014s
I20251023 02:04:25.407696 21972 tablet_copy_service.cc:140] P 02c53ba06cc3445a8534a84aa76b1c14: Received BeginTabletCopySession request for tablet 365d47bf062b4ebabb6d5f247e4e3e33 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:36488)
I20251023 02:04:25.408190 21972 tablet_copy_service.cc:161] P 02c53ba06cc3445a8534a84aa76b1c14: Beginning new tablet copy session on tablet 365d47bf062b4ebabb6d5f247e4e3e33 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:36488: session id = 1ab8b0b58147446d84d901b4ed2ff39e-365d47bf062b4ebabb6d5f247e4e3e33
I20251023 02:04:25.409703 22682 raft_consensus.cc:359] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:25.410490 22682 raft_consensus.cc:740] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:25.411011 22682 consensus_queue.cc:260] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:25.415447 21972 tablet_copy_source_session.cc:215] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:25.418642 22686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365d47bf062b4ebabb6d5f247e4e3e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:25.418888 22682 ts_tablet_manager.cc:1434] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20251023 02:04:25.420267 22649 heartbeater.cc:499] Master 127.20.81.62:41229 was elected leader, sending a full tablet report...
I20251023 02:04:25.421173 21972 tablet_copy_service.cc:342] P 02c53ba06cc3445a8534a84aa76b1c14: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-972d07041fbd4097854ef23c73f09c75 received from {username='slave'} at 127.0.0.1:36488
I20251023 02:04:25.421686 21972 tablet_copy_service.cc:434] P 02c53ba06cc3445a8534a84aa76b1c14: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-972d07041fbd4097854ef23c73f09c75 on tablet 972d07041fbd4097854ef23c73f09c75 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:25.439397 22686 tablet_copy_client.cc:806] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:25.440058 22686 tablet_copy_client.cc:670] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:25.443769 22686 tablet_copy_client.cc:538] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:25.450353 22686 tablet_bootstrap.cc:492] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:25.467836 22686 tablet_bootstrap.cc:492] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:25.468573 22686 tablet_bootstrap.cc:492] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:25.469185 22686 ts_tablet_manager.cc:1403] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20251023 02:04:25.471706 22686 raft_consensus.cc:359] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:25.472455 22686 raft_consensus.cc:740] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:25.472983 22686 consensus_queue.cc:260] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:25.475256 22686 ts_tablet_manager.cc:1434] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20251023 02:04:25.482146 21972 tablet_copy_service.cc:342] P 02c53ba06cc3445a8534a84aa76b1c14: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-365d47bf062b4ebabb6d5f247e4e3e33 received from {username='slave'} at 127.0.0.1:36488
I20251023 02:04:25.483160 21972 tablet_copy_service.cc:434] P 02c53ba06cc3445a8534a84aa76b1c14: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-365d47bf062b4ebabb6d5f247e4e3e33 on tablet 365d47bf062b4ebabb6d5f247e4e3e33 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:25.554150 22686 ts_tablet_manager.cc:933] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:25.555819 22686 tablet_copy_client.cc:323] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.2:40645
I20251023 02:04:25.561194 22682 ts_tablet_manager.cc:933] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:25.567368 22047 tablet_copy_service.cc:140] P 818907d63f1f4d84ba267ebe6ee837e0: Received BeginTabletCopySession request for tablet e32bb6c0622f44f8806f77eee0691856 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:55770)
I20251023 02:04:25.567951 22047 tablet_copy_service.cc:161] P 818907d63f1f4d84ba267ebe6ee837e0: Beginning new tablet copy session on tablet e32bb6c0622f44f8806f77eee0691856 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:55770: session id = 1ab8b0b58147446d84d901b4ed2ff39e-e32bb6c0622f44f8806f77eee0691856
I20251023 02:04:25.569403 22682 tablet_copy_client.cc:323] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.2:40645
I20251023 02:04:25.571583 22046 tablet_copy_service.cc:140] P 818907d63f1f4d84ba267ebe6ee837e0: Received BeginTabletCopySession request for tablet 774c010489aa4775bb6d636926900529 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:55770)
I20251023 02:04:25.572075 22046 tablet_copy_service.cc:161] P 818907d63f1f4d84ba267ebe6ee837e0: Beginning new tablet copy session on tablet 774c010489aa4775bb6d636926900529 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:55770: session id = 1ab8b0b58147446d84d901b4ed2ff39e-774c010489aa4775bb6d636926900529
I20251023 02:04:25.576277 22047 tablet_copy_source_session.cc:215] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:25.579072 22046 tablet_copy_source_session.cc:215] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:25.581954 22682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774c010489aa4775bb6d636926900529. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:25.583098 22686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32bb6c0622f44f8806f77eee0691856. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:25.598038 22682 tablet_copy_client.cc:806] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:25.598690 22682 tablet_copy_client.cc:670] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:25.598760 22686 tablet_copy_client.cc:806] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:25.599469 22686 tablet_copy_client.cc:670] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:25.602784 22686 tablet_copy_client.cc:538] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:25.602792 22682 tablet_copy_client.cc:538] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:25.611155 22686 tablet_bootstrap.cc:492] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:25.615260 22682 tablet_bootstrap.cc:492] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:25.631637 22686 tablet_bootstrap.cc:492] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:25.632551 22686 tablet_bootstrap.cc:492] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:25.633493 22686 ts_tablet_manager.cc:1403] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.001s
I20251023 02:04:25.636209 22686 raft_consensus.cc:359] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:25.637060 22686 raft_consensus.cc:740] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:25.637550 22682 tablet_bootstrap.cc:492] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:25.638473 22682 tablet_bootstrap.cc:492] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:25.637754 22686 consensus_queue.cc:260] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:25.639194 22682 ts_tablet_manager.cc:1403] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.001s
I20251023 02:04:25.640762 22686 ts_tablet_manager.cc:1434] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251023 02:04:25.651991 22047 tablet_copy_service.cc:342] P 818907d63f1f4d84ba267ebe6ee837e0: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-e32bb6c0622f44f8806f77eee0691856 received from {username='slave'} at 127.0.0.1:55770
I20251023 02:04:25.652395 22047 tablet_copy_service.cc:434] P 818907d63f1f4d84ba267ebe6ee837e0: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-e32bb6c0622f44f8806f77eee0691856 on tablet e32bb6c0622f44f8806f77eee0691856 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:25.654034 22682 raft_consensus.cc:359] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:25.654939 22682 raft_consensus.cc:740] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:25.655604 22682 consensus_queue.cc:260] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:25.660743 22682 ts_tablet_manager.cc:1434] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20251023 02:04:25.663487 22047 tablet_copy_service.cc:342] P 818907d63f1f4d84ba267ebe6ee837e0: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-774c010489aa4775bb6d636926900529 received from {username='slave'} at 127.0.0.1:55770
I20251023 02:04:25.663879 22047 tablet_copy_service.cc:434] P 818907d63f1f4d84ba267ebe6ee837e0: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-774c010489aa4775bb6d636926900529 on tablet 774c010489aa4775bb6d636926900529 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:25.925093 22624 raft_consensus.cc:1217] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:25.937695 22624 raft_consensus.cc:1217] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:25.973565 22624 raft_consensus.cc:1217] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:26.182548 22624 raft_consensus.cc:1217] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:26.478313 22698 raft_consensus.cc:1064] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:26.479605 22345 raft_consensus.cc:1064] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:26.480741 22698 consensus_queue.cc:237] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:26.481485 22345 consensus_queue.cc:237] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:26.492786 22624 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.494314 22701 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:26.496791 22624 raft_consensus.cc:1275] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.498032 22107 raft_consensus.cc:1275] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.498361 22695 consensus_queue.cc:1048] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:26.499400 22678 consensus_queue.cc:1048] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:26.526472 22697 raft_consensus.cc:1064] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:26.528606 22697 consensus_queue.cc:237] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:26.535547 22345 raft_consensus.cc:2955] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.537631 22030 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.539265 22698 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:26.540428 22107 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.545471 22033 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.547451 21959 raft_consensus.cc:1275] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.547336 22701 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:26.550215 22111 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.552372 22678 consensus_queue.cc:1048] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:26.555835 22624 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.560344 22697 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:26.550024 22698 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:26.564253 22698 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:26.566890 21959 raft_consensus.cc:2955] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.574456 22622 raft_consensus.cc:2955] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.590487 21860 catalog_manager.cc:5649] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:04:26.592665 22701 raft_consensus.cc:2955] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.594256 22111 raft_consensus.cc:2955] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.596077 22697 raft_consensus.cc:2955] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.599035 22624 raft_consensus.cc:2955] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.613423 22030 raft_consensus.cc:2955] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.613816 22623 raft_consensus.cc:2955] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.619537 22109 raft_consensus.cc:2955] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.623605 22107 raft_consensus.cc:2955] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.632826 22345 raft_consensus.cc:1064] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:26.635040 22345 consensus_queue.cc:237] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:26.641769 22033 raft_consensus.cc:2955] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.643957 22624 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.645485 21859 catalog_manager.cc:5649] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.646623 21959 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.647810 22109 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:26.648411 22695 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:26.649473 22678 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:26.650864 22523 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:26.623656 21858 catalog_manager.cc:5649] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:04:26.670686 22695 raft_consensus.cc:2955] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.674625 21957 raft_consensus.cc:2955] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.683998 22622 raft_consensus.cc:2955] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.687256 22109 raft_consensus.cc:2955] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.719247 21959 consensus_queue.cc:237] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:26.726470 22622 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:26.728708 22697 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:26.726322 21858 catalog_manager.cc:5649] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.731536 22109 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:26.733953 22697 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:26.736979 22702 raft_consensus.cc:2955] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.737986 22622 raft_consensus.cc:2955] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.742601 22109 raft_consensus.cc:2955] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.750797 21846 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 365d47bf062b4ebabb6d5f247e4e3e33 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:26.768836 21858 catalog_manager.cc:5649] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.772289 22033 consensus_queue.cc:237] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:26.779839 22020 tablet_service.cc:1552] Processing DeleteTablet for tablet 365d47bf062b4ebabb6d5f247e4e3e33 with delete_type TABLET_DATA_TOMBSTONED (TS 818907d63f1f4d84ba267ebe6ee837e0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38996
W20251023 02:04:26.781001 21846 catalog_manager.cc:5172] ChangeConfig:REMOVE_PEER RPC for tablet 365d47bf062b4ebabb6d5f247e4e3e33 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20251023 02:04:26.783803 22622 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:26.785151 22704 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:26.786553 22109 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:26.786484 21959 consensus_queue.cc:237] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:26.809677 22695 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:26.810020 22622 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:26.811789 22702 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:26.822263 22704 raft_consensus.cc:2955] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.828251 22107 raft_consensus.cc:2955] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.830684 22624 raft_consensus.cc:2955] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.834338 22035 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:26.835817 22697 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:26.837106 22724 tablet_replica.cc:333] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0: stopping tablet replica
I20251023 02:04:26.828372 22702 raft_consensus.cc:2955] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.840022 22035 raft_consensus.cc:2955] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.838236 22622 raft_consensus.cc:2955] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.850409 21846 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 972d07041fbd4097854ef23c73f09c75 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:26.854593 21858 catalog_manager.cc:5649] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:26.838148 22724 raft_consensus.cc:2243] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:26.863219 22724 raft_consensus.cc:2272] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:26.865383 21847 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 774c010489aa4775bb6d636926900529 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:26.868680 21948 tablet_service.cc:1552] Processing DeleteTablet for tablet 774c010489aa4775bb6d636926900529 with delete_type TABLET_DATA_TOMBSTONED (TS 02c53ba06cc3445a8534a84aa76b1c14 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53732
I20251023 02:04:26.868992 22724 ts_tablet_manager.cc:1916] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:26.874145 21859 catalog_manager.cc:5649] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:04:26.886266 22096 tablet_service.cc:1552] Processing DeleteTablet for tablet 972d07041fbd4097854ef23c73f09c75 with delete_type TABLET_DATA_TOMBSTONED (TS ca1362b9d9b34073bf35b6f6c9d1159a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39634
I20251023 02:04:26.886566 22724 ts_tablet_manager.cc:1929] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:26.887086 22724 log.cc:1199] T 365d47bf062b4ebabb6d5f247e4e3e33 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-1-root/wals/365d47bf062b4ebabb6d5f247e4e3e33
I20251023 02:04:26.893911 21847 catalog_manager.cc:4985] TS 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): tablet 365d47bf062b4ebabb6d5f247e4e3e33 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:26.903081 22725 tablet_replica.cc:333] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14: stopping tablet replica
I20251023 02:04:26.905485 22725 raft_consensus.cc:2243] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:26.906251 22725 raft_consensus.cc:2272] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:26.907757 22726 tablet_replica.cc:333] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a: stopping tablet replica
I20251023 02:04:26.908718 22726 raft_consensus.cc:2243] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:26.909276 22726 raft_consensus.cc:2272] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:26.912475 22726 ts_tablet_manager.cc:1916] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:26.913993 22725 ts_tablet_manager.cc:1916] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:26.930143 22725 ts_tablet_manager.cc:1929] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:26.930603 22725 log.cc:1199] T 774c010489aa4775bb6d636926900529 P 02c53ba06cc3445a8534a84aa76b1c14: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-0-root/wals/774c010489aa4775bb6d636926900529
I20251023 02:04:26.931773 22726 ts_tablet_manager.cc:1929] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:26.932178 21846 catalog_manager.cc:4985] TS 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): tablet 774c010489aa4775bb6d636926900529 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:26.932224 22726 log.cc:1199] T 972d07041fbd4097854ef23c73f09c75 P ca1362b9d9b34073bf35b6f6c9d1159a: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-2-root/wals/972d07041fbd4097854ef23c73f09c75
I20251023 02:04:26.935495 21844 catalog_manager.cc:4985] TS ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): tablet 972d07041fbd4097854ef23c73f09c75 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:27.004123 22030 consensus_queue.cc:237] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:27.008342 21959 raft_consensus.cc:1275] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:27.008409 22622 raft_consensus.cc:1275] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:27.009392 22523 consensus_queue.cc:1048] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:27.010093 22704 consensus_queue.cc:1048] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:27.016723 22523 raft_consensus.cc:2955] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:27.019121 22622 raft_consensus.cc:2955] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:27.022379 21958 raft_consensus.cc:2955] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:27.034174 21859 catalog_manager.cc:5649] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:04:27.039067 21847 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet e32bb6c0622f44f8806f77eee0691856 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:27.046422 22096 tablet_service.cc:1552] Processing DeleteTablet for tablet e32bb6c0622f44f8806f77eee0691856 with delete_type TABLET_DATA_TOMBSTONED (TS ca1362b9d9b34073bf35b6f6c9d1159a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39634
I20251023 02:04:27.047493 22726 tablet_replica.cc:333] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a: stopping tablet replica
I20251023 02:04:27.048401 22726 raft_consensus.cc:2243] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:27.049047 22726 raft_consensus.cc:2272] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:27.053270 22726 ts_tablet_manager.cc:1916] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:27.064663 22726 ts_tablet_manager.cc:1929] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:27.064965 22726 log.cc:1199] T e32bb6c0622f44f8806f77eee0691856 P ca1362b9d9b34073bf35b6f6c9d1159a: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-2-root/wals/e32bb6c0622f44f8806f77eee0691856
I20251023 02:04:27.066130 21844 catalog_manager.cc:4985] TS ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): tablet e32bb6c0622f44f8806f77eee0691856 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:28.119307 22107 consensus_queue.cc:237] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:28.125365 21959 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:28.125952 22035 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:28.127126 22730 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:28.127970 22403 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:28.137241 22403 raft_consensus.cc:2955] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
W20251023 02:04:28.147023 22075 consensus_peers.cc:597] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0325fa135bce480eaaa5faf74daf54f8. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:28.147636 22035 raft_consensus.cc:2955] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.155285 22107 consensus_queue.cc:237] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:28.156335 21960 raft_consensus.cc:2955] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.159680 21859 catalog_manager.cc:5649] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:28.166059 22035 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:28.167590 22730 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:28.168565 21958 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251023 02:04:28.169988 22075 consensus_peers.cc:597] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97dae00618e34f58962a1dfbaa38fcdc. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:28.170956 22730 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:28.212513 22210 raft_consensus.cc:2955] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.214756 21960 raft_consensus.cc:2955] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.212636 22033 raft_consensus.cc:2955] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.235898 21858 catalog_manager.cc:5649] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.239972 22030 consensus_queue.cc:237] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
W20251023 02:04:28.251341 22001 consensus_peers.cc:597] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 816ef481cee84b259bc409b45a81baf1. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:28.251590 22109 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:28.254251 21960 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:28.253428 22523 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:28.256672 22704 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:28.270612 22107 raft_consensus.cc:2955] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.269435 22744 raft_consensus.cc:2955] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.277832 21960 raft_consensus.cc:2955] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.288499 21860 catalog_manager.cc:5649] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.290575 21960 consensus_queue.cc:237] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:28.302011 22107 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251023 02:04:28.302201 21926 consensus_peers.cc:597] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d2df37149b145b89207c30e1a0360b0. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:28.303331 22702 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:28.308040 22030 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:28.309365 22697 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:28.311416 22697 raft_consensus.cc:2955] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.315062 22107 raft_consensus.cc:2955] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.334192 22033 raft_consensus.cc:2955] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.336639 21858 catalog_manager.cc:5649] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:28.705058 22751 ts_tablet_manager.cc:933] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:28.707166 22751 tablet_copy_client.cc:323] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.3:33899
I20251023 02:04:28.730666 22121 tablet_copy_service.cc:140] P ca1362b9d9b34073bf35b6f6c9d1159a: Received BeginTabletCopySession request for tablet 0325fa135bce480eaaa5faf74daf54f8 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:56894)
I20251023 02:04:28.731290 22121 tablet_copy_service.cc:161] P ca1362b9d9b34073bf35b6f6c9d1159a: Beginning new tablet copy session on tablet 0325fa135bce480eaaa5faf74daf54f8 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:56894: session id = 1ab8b0b58147446d84d901b4ed2ff39e-0325fa135bce480eaaa5faf74daf54f8
I20251023 02:04:28.734573 22755 ts_tablet_manager.cc:933] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:28.739652 22121 tablet_copy_source_session.cc:215] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:28.742892 22751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0325fa135bce480eaaa5faf74daf54f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:28.743311 22755 tablet_copy_client.cc:323] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.2:40645
I20251023 02:04:28.744884 22047 tablet_copy_service.cc:140] P 818907d63f1f4d84ba267ebe6ee837e0: Received BeginTabletCopySession request for tablet 816ef481cee84b259bc409b45a81baf1 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:55770)
I20251023 02:04:28.745399 22047 tablet_copy_service.cc:161] P 818907d63f1f4d84ba267ebe6ee837e0: Beginning new tablet copy session on tablet 816ef481cee84b259bc409b45a81baf1 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:55770: session id = 1ab8b0b58147446d84d901b4ed2ff39e-816ef481cee84b259bc409b45a81baf1
I20251023 02:04:28.754561 22047 tablet_copy_source_session.cc:215] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:28.763016 22755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 816ef481cee84b259bc409b45a81baf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:28.766530 22758 ts_tablet_manager.cc:933] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499)
I20251023 02:04:28.765169 22751 tablet_copy_client.cc:806] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:28.767259 22751 tablet_copy_client.cc:670] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:28.775576 22758 tablet_copy_client.cc:323] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.1:41499
I20251023 02:04:28.777251 21972 tablet_copy_service.cc:140] P 02c53ba06cc3445a8534a84aa76b1c14: Received BeginTabletCopySession request for tablet 0d2df37149b145b89207c30e1a0360b0 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:36488)
I20251023 02:04:28.777792 21972 tablet_copy_service.cc:161] P 02c53ba06cc3445a8534a84aa76b1c14: Beginning new tablet copy session on tablet 0d2df37149b145b89207c30e1a0360b0 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:36488: session id = 1ab8b0b58147446d84d901b4ed2ff39e-0d2df37149b145b89207c30e1a0360b0
I20251023 02:04:28.790443 22755 tablet_copy_client.cc:806] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:28.790951 22757 ts_tablet_manager.cc:933] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:28.791045 22755 tablet_copy_client.cc:670] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:28.793462 22757 tablet_copy_client.cc:323] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.3:33899
I20251023 02:04:28.793728 21972 tablet_copy_source_session.cc:215] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:28.795137 22755 tablet_copy_client.cc:538] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:28.796681 22758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2df37149b145b89207c30e1a0360b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:28.797230 22121 tablet_copy_service.cc:140] P ca1362b9d9b34073bf35b6f6c9d1159a: Received BeginTabletCopySession request for tablet 97dae00618e34f58962a1dfbaa38fcdc from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:56894)
I20251023 02:04:28.797788 22121 tablet_copy_service.cc:161] P ca1362b9d9b34073bf35b6f6c9d1159a: Beginning new tablet copy session on tablet 97dae00618e34f58962a1dfbaa38fcdc from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:56894: session id = 1ab8b0b58147446d84d901b4ed2ff39e-97dae00618e34f58962a1dfbaa38fcdc
I20251023 02:04:28.800586 22751 tablet_copy_client.cc:538] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:28.804061 22755 tablet_bootstrap.cc:492] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:28.807021 22121 tablet_copy_source_session.cc:215] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:28.823169 22757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97dae00618e34f58962a1dfbaa38fcdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:28.831543 22751 tablet_bootstrap.cc:492] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:28.837509 22758 tablet_copy_client.cc:806] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:28.838259 22758 tablet_copy_client.cc:670] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:28.844017 22758 tablet_copy_client.cc:538] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:28.847533 22755 tablet_bootstrap.cc:492] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:28.848479 22755 tablet_bootstrap.cc:492] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:28.849284 22755 ts_tablet_manager.cc:1403] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.045s user 0.017s sys 0.015s
I20251023 02:04:28.855700 22757 tablet_copy_client.cc:806] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:28.856387 22757 tablet_copy_client.cc:670] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:28.856138 22755 raft_consensus.cc:359] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:28.856964 22755 raft_consensus.cc:740] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:28.860463 22758 tablet_bootstrap.cc:492] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:28.864451 22757 tablet_copy_client.cc:538] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:28.864157 22755 consensus_queue.cc:260] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:28.878456 22757 tablet_bootstrap.cc:492] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:28.884366 22751 tablet_bootstrap.cc:492] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:28.885259 22751 tablet_bootstrap.cc:492] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:28.886018 22751 ts_tablet_manager.cc:1403] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.055s user 0.024s sys 0.004s
I20251023 02:04:28.888172 22755 ts_tablet_manager.cc:1434] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20251023 02:04:28.889242 22751 raft_consensus.cc:359] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:28.890300 22047 tablet_copy_service.cc:342] P 818907d63f1f4d84ba267ebe6ee837e0: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-816ef481cee84b259bc409b45a81baf1 received from {username='slave'} at 127.0.0.1:55770
I20251023 02:04:28.890810 22047 tablet_copy_service.cc:434] P 818907d63f1f4d84ba267ebe6ee837e0: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-816ef481cee84b259bc409b45a81baf1 on tablet 816ef481cee84b259bc409b45a81baf1 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:28.893651 22751 raft_consensus.cc:740] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:28.894424 22751 consensus_queue.cc:260] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:28.896824 22758 tablet_bootstrap.cc:492] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:28.896888 22751 ts_tablet_manager.cc:1434] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20251023 02:04:28.897863 22758 tablet_bootstrap.cc:492] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:28.899489 22121 tablet_copy_service.cc:342] P ca1362b9d9b34073bf35b6f6c9d1159a: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-0325fa135bce480eaaa5faf74daf54f8 received from {username='slave'} at 127.0.0.1:56894
I20251023 02:04:28.899921 22121 tablet_copy_service.cc:434] P ca1362b9d9b34073bf35b6f6c9d1159a: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-0325fa135bce480eaaa5faf74daf54f8 on tablet 0325fa135bce480eaaa5faf74daf54f8 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:28.905807 22758 ts_tablet_manager.cc:1403] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.046s user 0.018s sys 0.005s
I20251023 02:04:28.908787 22758 raft_consensus.cc:359] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:28.909668 22758 raft_consensus.cc:740] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:28.910334 22758 consensus_queue.cc:260] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:28.912824 22758 ts_tablet_manager.cc:1434] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251023 02:04:28.916352 21972 tablet_copy_service.cc:342] P 02c53ba06cc3445a8534a84aa76b1c14: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-0d2df37149b145b89207c30e1a0360b0 received from {username='slave'} at 127.0.0.1:36488
I20251023 02:04:28.916747 21972 tablet_copy_service.cc:434] P 02c53ba06cc3445a8534a84aa76b1c14: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-0d2df37149b145b89207c30e1a0360b0 on tablet 0d2df37149b145b89207c30e1a0360b0 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:28.927738 22757 tablet_bootstrap.cc:492] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:28.928727 22757 tablet_bootstrap.cc:492] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:28.929451 22757 ts_tablet_manager.cc:1403] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.017s
I20251023 02:04:28.932670 22757 raft_consensus.cc:359] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:28.933609 22757 raft_consensus.cc:740] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:28.934311 22757 consensus_queue.cc:260] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:28.936584 22757 ts_tablet_manager.cc:1434] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251023 02:04:28.939108 22121 tablet_copy_service.cc:342] P ca1362b9d9b34073bf35b6f6c9d1159a: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-97dae00618e34f58962a1dfbaa38fcdc received from {username='slave'} at 127.0.0.1:56894
I20251023 02:04:28.939473 22121 tablet_copy_service.cc:434] P ca1362b9d9b34073bf35b6f6c9d1159a: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-97dae00618e34f58962a1dfbaa38fcdc on tablet 97dae00618e34f58962a1dfbaa38fcdc with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:29.152838 22622 raft_consensus.cc:1217] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:29.182353 22622 raft_consensus.cc:1217] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:29.290277 22622 raft_consensus.cc:1217] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:29.325038 22622 raft_consensus.cc:1217] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:29.602910 22766 raft_consensus.cc:1064] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:29.605273 22766 consensus_queue.cc:237] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:29.611243 22622 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.612141 21958 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.612928 22035 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.613387 22767 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:29.614279 22210 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:29.615026 22766 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:29.628440 22766 raft_consensus.cc:2955] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.639047 22035 raft_consensus.cc:2955] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.640852 22622 raft_consensus.cc:2955] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.655514 21958 raft_consensus.cc:2955] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.659709 21859 catalog_manager.cc:5649] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.689714 22107 consensus_queue.cc:237] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:29.697583 22035 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:29.699038 22767 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:29.708678 22622 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:29.709545 22766 raft_consensus.cc:2955] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.711261 22767 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:29.715111 22622 raft_consensus.cc:2955] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.717289 22767 raft_consensus.cc:1064] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:29.719157 22035 raft_consensus.cc:2955] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.720023 22767 consensus_queue.cc:237] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:29.734953 21857 catalog_manager.cc:5649] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.743839 22035 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.746237 21960 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.747484 21948 tablet_service.cc:1552] Processing DeleteTablet for tablet 97dae00618e34f58962a1dfbaa38fcdc with delete_type TABLET_DATA_TOMBSTONED (TS 02c53ba06cc3445a8534a84aa76b1c14 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53732
I20251023 02:04:29.749945 22780 tablet_replica.cc:333] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14: stopping tablet replica
I20251023 02:04:29.752193 22780 raft_consensus.cc:2243] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:29.753767 22778 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:29.754683 22622 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.754882 21844 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 97dae00618e34f58962a1dfbaa38fcdc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:29.755851 22780 raft_consensus.cc:2272] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:29.758123 22210 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:29.759917 22778 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:29.788635 22780 ts_tablet_manager.cc:1916] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:29.793202 22778 raft_consensus.cc:2955] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.794759 22035 raft_consensus.cc:2955] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.804773 22622 raft_consensus.cc:2955] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.818076 21857 catalog_manager.cc:5649] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.821741 22780 ts_tablet_manager.cc:1929] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:29.828099 22780 log.cc:1199] T 97dae00618e34f58962a1dfbaa38fcdc P 02c53ba06cc3445a8534a84aa76b1c14: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-0-root/wals/97dae00618e34f58962a1dfbaa38fcdc
I20251023 02:04:29.829720 21846 catalog_manager.cc:4985] TS 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): tablet 97dae00618e34f58962a1dfbaa38fcdc (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:29.836575 21958 raft_consensus.cc:2955] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.849042 22704 raft_consensus.cc:1064] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:29.851202 22704 consensus_queue.cc:237] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:29.875059 22110 consensus_queue.cc:237] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:29.879789 22622 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.881650 22787 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:29.882225 21959 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.883072 22111 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.889007 22749 raft_consensus.cc:1064] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:29.885944 22768 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20251023 02:04:29.891506 22749 consensus_queue.cc:237] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:29.894594 22768 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:29.904482 22622 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.905010 22033 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.905792 22697 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:29.912487 21958 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:29.913802 22778 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:04:29.916410 22622 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:29.919514 22766 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:04:29.923238 22702 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:29.907271 22107 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:29.924211 22781 raft_consensus.cc:2955] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.929260 22768 raft_consensus.cc:2955] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.933915 21957 raft_consensus.cc:2955] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.935691 22749 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:29.947484 21844 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 0325fa135bce480eaaa5faf74daf54f8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:29.959477 22109 raft_consensus.cc:2955] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.965677 22622 raft_consensus.cc:2955] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.977337 22702 raft_consensus.cc:2955] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.978567 22107 raft_consensus.cc:2955] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:29.989408 21860 catalog_manager.cc:5649] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.008688 22033 raft_consensus.cc:2955] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.010939 22020 tablet_service.cc:1552] Processing DeleteTablet for tablet 0325fa135bce480eaaa5faf74daf54f8 with delete_type TABLET_DATA_TOMBSTONED (TS 818907d63f1f4d84ba267ebe6ee837e0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38996
I20251023 02:04:30.007839 22624 raft_consensus.cc:2955] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.037087 22622 raft_consensus.cc:2955] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.039374 22797 tablet_replica.cc:333] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0: stopping tablet replica
I20251023 02:04:30.046241 21957 raft_consensus.cc:2955] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.058346 21858 catalog_manager.cc:5649] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:30.060671 21860 catalog_manager.cc:5649] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:04:30.070288 22797 raft_consensus.cc:2243] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:30.071067 22797 raft_consensus.cc:2272] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:30.074388 22797 ts_tablet_manager.cc:1916] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:30.081492 22033 consensus_queue.cc:237] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:30.094090 22107 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:30.095585 22787 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:30.096447 22622 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:30.097735 22523 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:30.104826 22523 raft_consensus.cc:2955] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.106679 22624 raft_consensus.cc:2955] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.106765 22109 raft_consensus.cc:2955] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.126332 22797 ts_tablet_manager.cc:1929] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:30.126982 21847 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 816ef481cee84b259bc409b45a81baf1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:30.126920 22797 log.cc:1199] T 0325fa135bce480eaaa5faf74daf54f8 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-1-root/wals/0325fa135bce480eaaa5faf74daf54f8
I20251023 02:04:30.149314 21847 catalog_manager.cc:4985] TS 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): tablet 0325fa135bce480eaaa5faf74daf54f8 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:30.151621 21959 consensus_queue.cc:237] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:30.154078 21859 catalog_manager.cc:5649] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.159240 22107 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:30.161052 22624 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:30.163417 22749 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:30.165072 22796 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:04:30.170511 22702 raft_consensus.cc:2955] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.171725 21948 tablet_service.cc:1552] Processing DeleteTablet for tablet 816ef481cee84b259bc409b45a81baf1 with delete_type TABLET_DATA_TOMBSTONED (TS 02c53ba06cc3445a8534a84aa76b1c14 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53732
I20251023 02:04:30.173184 22780 tablet_replica.cc:333] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14: stopping tablet replica
I20251023 02:04:30.174036 22780 raft_consensus.cc:2243] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:30.174566 22780 raft_consensus.cc:2272] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:30.178107 21846 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 0d2df37149b145b89207c30e1a0360b0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:30.178294 22780 ts_tablet_manager.cc:1916] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:30.179069 22624 raft_consensus.cc:2955] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.183869 22107 raft_consensus.cc:2955] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.194183 21859 catalog_manager.cc:5649] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:30.207435 22020 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d2df37149b145b89207c30e1a0360b0 with delete_type TABLET_DATA_TOMBSTONED (TS 818907d63f1f4d84ba267ebe6ee837e0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38996
I20251023 02:04:30.208273 22797 tablet_replica.cc:333] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0: stopping tablet replica
I20251023 02:04:30.208840 22780 ts_tablet_manager.cc:1929] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:30.209306 22797 raft_consensus.cc:2243] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:30.210062 22780 log.cc:1199] T 816ef481cee84b259bc409b45a81baf1 P 02c53ba06cc3445a8534a84aa76b1c14: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-0-root/wals/816ef481cee84b259bc409b45a81baf1
I20251023 02:04:30.210127 22797 raft_consensus.cc:2272] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:30.213524 21846 catalog_manager.cc:4985] TS 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): tablet 816ef481cee84b259bc409b45a81baf1 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:30.213618 22797 ts_tablet_manager.cc:1916] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:30.227466 22797 ts_tablet_manager.cc:1929] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:30.227952 22797 log.cc:1199] T 0d2df37149b145b89207c30e1a0360b0 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-1-root/wals/0d2df37149b145b89207c30e1a0360b0
I20251023 02:04:30.229178 21847 catalog_manager.cc:4985] TS 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): tablet 0d2df37149b145b89207c30e1a0360b0 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:31.251036 21959 consensus_queue.cc:237] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:31.257365 22036 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251023 02:04:31.257625 21926 consensus_peers.cc:597] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2da2ca0f34944635a558b8678ecc206f. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:31.257970 22111 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:31.258838 22697 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:31.259502 22799 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:31.266603 22697 raft_consensus.cc:2955] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.271940 22111 raft_consensus.cc:2955] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.280635 22036 raft_consensus.cc:2955] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.297086 21860 catalog_manager.cc:5649] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.302485 21959 consensus_queue.cc:237] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:31.309798 22109 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251023 02:04:31.312122 21926 consensus_peers.cc:597] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe0e348fc464e36be33082ddcdba159. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:31.312624 22036 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:31.312565 22799 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:31.314303 22803 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:31.323297 22799 raft_consensus.cc:2955] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.324992 22111 raft_consensus.cc:2955] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.341068 22030 raft_consensus.cc:2955] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.348030 21858 catalog_manager.cc:5649] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.352483 22030 consensus_queue.cc:237] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
W20251023 02:04:31.375988 22001 consensus_peers.cc:597] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe5d794166cc4fbb87c6b918fa26b592. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:31.376456 22109 raft_consensus.cc:1275] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:31.377816 21959 raft_consensus.cc:1275] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:31.378451 22523 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:31.379869 22523 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:31.403065 22801 raft_consensus.cc:2955] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.406143 22109 raft_consensus.cc:2955] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.407207 21957 raft_consensus.cc:2955] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.416378 22111 consensus_queue.cc:237] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:31.424086 21859 catalog_manager.cc:5649] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251023 02:04:31.431196 22075 consensus_peers.cc:597] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0a3abec901e4c2eba45233d075d9263. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:31.431639 22035 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:31.432909 22815 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:31.433291 21957 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:31.434365 22778 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:31.449697 22778 raft_consensus.cc:2955] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.451275 21957 raft_consensus.cc:2955] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.458196 22035 raft_consensus.cc:2955] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:31.479773 21860 catalog_manager.cc:5649] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:31.756443 22824 ts_tablet_manager.cc:933] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499)
I20251023 02:04:31.758677 22824 tablet_copy_client.cc:323] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.1:41499
I20251023 02:04:31.794986 21972 tablet_copy_service.cc:140] P 02c53ba06cc3445a8534a84aa76b1c14: Received BeginTabletCopySession request for tablet ffe0e348fc464e36be33082ddcdba159 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:36488)
I20251023 02:04:31.796554 21972 tablet_copy_service.cc:161] P 02c53ba06cc3445a8534a84aa76b1c14: Beginning new tablet copy session on tablet ffe0e348fc464e36be33082ddcdba159 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:36488: session id = 1ab8b0b58147446d84d901b4ed2ff39e-ffe0e348fc464e36be33082ddcdba159
I20251023 02:04:31.829947 21972 tablet_copy_source_session.cc:215] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:31.866389 22828 ts_tablet_manager.cc:933] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499)
I20251023 02:04:31.888577 22828 tablet_copy_client.cc:323] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.1:41499
I20251023 02:04:31.898598 22824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe0e348fc464e36be33082ddcdba159. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:31.917847 21972 tablet_copy_service.cc:140] P 02c53ba06cc3445a8534a84aa76b1c14: Received BeginTabletCopySession request for tablet 2da2ca0f34944635a558b8678ecc206f from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:36488)
I20251023 02:04:31.918542 21972 tablet_copy_service.cc:161] P 02c53ba06cc3445a8534a84aa76b1c14: Beginning new tablet copy session on tablet 2da2ca0f34944635a558b8678ecc206f from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:36488: session id = 1ab8b0b58147446d84d901b4ed2ff39e-2da2ca0f34944635a558b8678ecc206f
I20251023 02:04:31.945010 21972 tablet_copy_source_session.cc:215] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:31.950872 22824 tablet_copy_client.cc:806] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:31.951481 22830 ts_tablet_manager.cc:933] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:31.951838 22824 tablet_copy_client.cc:670] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:31.958636 22828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da2ca0f34944635a558b8678ecc206f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:31.962904 22832 ts_tablet_manager.cc:933] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:31.965404 22830 tablet_copy_client.cc:323] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.3:33899
I20251023 02:04:31.967525 22121 tablet_copy_service.cc:140] P ca1362b9d9b34073bf35b6f6c9d1159a: Received BeginTabletCopySession request for tablet c0a3abec901e4c2eba45233d075d9263 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:56894)
I20251023 02:04:31.968135 22121 tablet_copy_service.cc:161] P ca1362b9d9b34073bf35b6f6c9d1159a: Beginning new tablet copy session on tablet c0a3abec901e4c2eba45233d075d9263 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:56894: session id = 1ab8b0b58147446d84d901b4ed2ff39e-c0a3abec901e4c2eba45233d075d9263
I20251023 02:04:31.969095 22824 tablet_copy_client.cc:538] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:32.007041 22824 tablet_bootstrap.cc:492] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:32.011926 22121 tablet_copy_source_session.cc:215] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:32.021379 22832 tablet_copy_client.cc:323] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.2:40645
I20251023 02:04:32.023885 22047 tablet_copy_service.cc:140] P 818907d63f1f4d84ba267ebe6ee837e0: Received BeginTabletCopySession request for tablet fe5d794166cc4fbb87c6b918fa26b592 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:55770)
I20251023 02:04:32.024469 22047 tablet_copy_service.cc:161] P 818907d63f1f4d84ba267ebe6ee837e0: Beginning new tablet copy session on tablet fe5d794166cc4fbb87c6b918fa26b592 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:55770: session id = 1ab8b0b58147446d84d901b4ed2ff39e-fe5d794166cc4fbb87c6b918fa26b592
I20251023 02:04:32.029646 22830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0a3abec901e4c2eba45233d075d9263. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:32.036995 22047 tablet_copy_source_session.cc:215] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:32.037706 22828 tablet_copy_client.cc:806] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:32.038393 22828 tablet_copy_client.cc:670] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:32.041465 22832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe5d794166cc4fbb87c6b918fa26b592. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:32.044368 22828 tablet_copy_client.cc:538] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:32.068698 22830 tablet_copy_client.cc:806] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:32.069468 22830 tablet_copy_client.cc:670] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:32.074213 22830 tablet_copy_client.cc:538] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:32.074479 22832 tablet_copy_client.cc:806] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:32.075083 22832 tablet_copy_client.cc:670] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:32.083861 22828 tablet_bootstrap.cc:492] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:32.082379 22832 tablet_copy_client.cc:538] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:32.094652 22824 tablet_bootstrap.cc:492] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:32.096129 22832 tablet_bootstrap.cc:492] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:32.098191 22824 tablet_bootstrap.cc:492] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:32.098929 22824 ts_tablet_manager.cc:1403] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.092s user 0.040s sys 0.008s
I20251023 02:04:32.096781 22830 tablet_bootstrap.cc:492] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:32.117746 22824 raft_consensus.cc:359] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:32.119254 22824 raft_consensus.cc:740] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:32.123062 22824 consensus_queue.cc:260] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:32.132462 22828 tablet_bootstrap.cc:492] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:32.133630 22828 tablet_bootstrap.cc:492] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:32.134332 22828 ts_tablet_manager.cc:1403] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.051s user 0.019s sys 0.005s
I20251023 02:04:32.137223 22832 tablet_bootstrap.cc:492] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:32.137199 22828 raft_consensus.cc:359] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:32.138195 22828 raft_consensus.cc:740] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:32.138336 22832 tablet_bootstrap.cc:492] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:32.140240 22832 ts_tablet_manager.cc:1403] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.044s user 0.026s sys 0.014s
I20251023 02:04:32.140167 22828 consensus_queue.cc:260] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:32.143584 22824 ts_tablet_manager.cc:1434] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.044s user 0.006s sys 0.006s
I20251023 02:04:32.144551 22832 raft_consensus.cc:359] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:32.145650 22832 raft_consensus.cc:740] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:32.146347 22832 consensus_queue.cc:260] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:32.149003 22832 ts_tablet_manager.cc:1434] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251023 02:04:32.149478 21971 tablet_copy_service.cc:342] P 02c53ba06cc3445a8534a84aa76b1c14: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-ffe0e348fc464e36be33082ddcdba159 received from {username='slave'} at 127.0.0.1:36488
I20251023 02:04:32.149988 21971 tablet_copy_service.cc:434] P 02c53ba06cc3445a8534a84aa76b1c14: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-ffe0e348fc464e36be33082ddcdba159 on tablet ffe0e348fc464e36be33082ddcdba159 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:32.151311 22047 tablet_copy_service.cc:342] P 818907d63f1f4d84ba267ebe6ee837e0: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-fe5d794166cc4fbb87c6b918fa26b592 received from {username='slave'} at 127.0.0.1:55770
I20251023 02:04:32.152042 22047 tablet_copy_service.cc:434] P 818907d63f1f4d84ba267ebe6ee837e0: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-fe5d794166cc4fbb87c6b918fa26b592 on tablet fe5d794166cc4fbb87c6b918fa26b592 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:32.158108 22828 ts_tablet_manager.cc:1434] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.023s user 0.007s sys 0.004s
I20251023 02:04:32.159744 21971 tablet_copy_service.cc:342] P 02c53ba06cc3445a8534a84aa76b1c14: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-2da2ca0f34944635a558b8678ecc206f received from {username='slave'} at 127.0.0.1:36488
I20251023 02:04:32.160132 21971 tablet_copy_service.cc:434] P 02c53ba06cc3445a8534a84aa76b1c14: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-2da2ca0f34944635a558b8678ecc206f on tablet 2da2ca0f34944635a558b8678ecc206f with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:32.161273 22830 tablet_bootstrap.cc:492] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:32.162230 22830 tablet_bootstrap.cc:492] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:32.162940 22830 ts_tablet_manager.cc:1403] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.066s user 0.024s sys 0.004s
I20251023 02:04:32.165171 22830 raft_consensus.cc:359] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:32.165884 22830 raft_consensus.cc:740] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:32.166515 22830 consensus_queue.cc:260] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:32.168807 22830 ts_tablet_manager.cc:1434] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251023 02:04:32.170519 22121 tablet_copy_service.cc:342] P ca1362b9d9b34073bf35b6f6c9d1159a: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-c0a3abec901e4c2eba45233d075d9263 received from {username='slave'} at 127.0.0.1:56894
I20251023 02:04:32.170931 22121 tablet_copy_service.cc:434] P ca1362b9d9b34073bf35b6f6c9d1159a: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-c0a3abec901e4c2eba45233d075d9263 on tablet c0a3abec901e4c2eba45233d075d9263 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:32.202652 22622 raft_consensus.cc:1217] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:32.238581 22622 raft_consensus.cc:1217] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:32.303407 22622 raft_consensus.cc:1217] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:32.358085 22622 raft_consensus.cc:1217] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:32.798902 22827 raft_consensus.cc:1064] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:32.801543 22827 consensus_queue.cc:237] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:32.816480 22622 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:32.818275 22111 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:32.820164 22697 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:32.821929 22827 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:32.823094 22030 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:32.824402 22827 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:32.845290 22778 raft_consensus.cc:1064] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:32.848224 22778 consensus_queue.cc:237] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:32.848968 22697 raft_consensus.cc:2955] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:32.854290 22846 raft_consensus.cc:1064] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:32.855175 22111 raft_consensus.cc:2955] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:32.857637 22846 consensus_queue.cc:237] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:32.864485 22035 raft_consensus.cc:2955] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:32.872244 21860 catalog_manager.cc:5649] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:32.878645 22622 raft_consensus.cc:2955] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:32.959232 22622 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:32.960453 22030 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:32.973759 22107 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:32.972681 22846 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20251023 02:04:32.967339 22851 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20251023 02:04:32.982173 22622 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:32.983718 22851 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:32.986177 22801 raft_consensus.cc:1064] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
W20251023 02:04:32.994004 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 56283.7us
I20251023 02:04:33.006441 22842 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:33.015316 22030 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:32.994973 22801 consensus_queue.cc:237] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:33.017490 22861 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:33.029222 21957 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:33.032528 22856 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251023 02:04:33.040910 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 60836.3us
I20251023 02:04:33.064481 22859 raft_consensus.cc:2955] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.069613 22861 raft_consensus.cc:2955] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.071388 22107 raft_consensus.cc:2955] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.096333 22036 raft_consensus.cc:2955] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.103158 22623 raft_consensus.cc:1275] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:33.104959 22852 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:33.111923 22109 raft_consensus.cc:1275] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:33.116916 22704 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:33.118765 22623 raft_consensus.cc:2955] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.128577 22035 raft_consensus.cc:2955] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.139950 21957 raft_consensus.cc:1275] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:33.141409 22852 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:33.143132 21958 consensus_queue.cc:237] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:33.131338 21960 raft_consensus.cc:2955] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
W20251023 02:04:33.146950 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 50276.4us
I20251023 02:04:33.162185 22621 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:33.169224 22860 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20251023 02:04:33.173939 22036 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:33.177729 22622 raft_consensus.cc:2955] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.188206 21859 catalog_manager.cc:5649] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:33.195154 22861 raft_consensus.cc:2955] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.171738 21858 catalog_manager.cc:5649] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:04:33.202318 22621 raft_consensus.cc:2955] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.248736 22842 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:33.257498 21846 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 2da2ca0f34944635a558b8678ecc206f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:33.262686 21960 consensus_queue.cc:237] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:33.268752 22035 raft_consensus.cc:2955] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.269703 22111 consensus_queue.cc:237] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:33.273784 22624 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:33.277407 22842 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20251023 02:04:33.284017 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 77729.2us
I20251023 02:04:33.284700 22107 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:33.279743 21860 catalog_manager.cc:5649] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.281276 22852 raft_consensus.cc:2955] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.289183 22621 raft_consensus.cc:2955] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.298624 22030 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:33.301048 22856 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:33.301471 22111 raft_consensus.cc:2955] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.309074 22842 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:33.311266 22624 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:33.312857 22856 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:04:33.315269 22851 raft_consensus.cc:2955] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.319309 22860 raft_consensus.cc:2955] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.321930 21958 raft_consensus.cc:2955] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.329258 21846 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet ffe0e348fc464e36be33082ddcdba159 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:33.330267 22621 raft_consensus.cc:2955] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.330883 22111 raft_consensus.cc:2955] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.334386 22096 tablet_service.cc:1552] Processing DeleteTablet for tablet 2da2ca0f34944635a558b8678ecc206f with delete_type TABLET_DATA_TOMBSTONED (TS ca1362b9d9b34073bf35b6f6c9d1159a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39634
I20251023 02:04:33.351850 21857 catalog_manager.cc:5649] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.354457 21856 catalog_manager.cc:5649] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:33.358290 22035 raft_consensus.cc:2955] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.357280 22622 raft_consensus.cc:2955] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.371697 22879 tablet_replica.cc:333] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a: stopping tablet replica
I20251023 02:04:33.372920 22879 raft_consensus.cc:2243] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:33.371747 21844 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet c0a3abec901e4c2eba45233d075d9263 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:33.374327 22879 raft_consensus.cc:2272] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:33.375603 22020 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe0e348fc464e36be33082ddcdba159 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:38996
I20251023 02:04:33.391677 21858 catalog_manager.cc:5649] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.379531 22879 ts_tablet_manager.cc:1916] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:33.406781 22880 tablet_replica.cc:333] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0: stopping tablet replica
I20251023 02:04:33.413395 21948 tablet_service.cc:1552] Processing DeleteTablet for tablet c0a3abec901e4c2eba45233d075d9263 with delete_type TABLET_DATA_TOMBSTONED (TS 02c53ba06cc3445a8534a84aa76b1c14 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53732
I20251023 02:04:33.414361 22879 ts_tablet_manager.cc:1929] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:33.414976 22879 log.cc:1199] T 2da2ca0f34944635a558b8678ecc206f P ca1362b9d9b34073bf35b6f6c9d1159a: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-2-root/wals/2da2ca0f34944635a558b8678ecc206f
I20251023 02:04:33.416642 21844 catalog_manager.cc:4985] TS ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): tablet 2da2ca0f34944635a558b8678ecc206f (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:33.434695 22033 consensus_queue.cc:237] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:33.443275 21958 raft_consensus.cc:1275] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:33.445240 22876 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:04:33.446952 22622 raft_consensus.cc:1275] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:33.452330 22704 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:33.452067 22878 raft_consensus.cc:2955] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.460826 22622 raft_consensus.cc:2955] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.464902 21958 raft_consensus.cc:2955] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.482061 21857 catalog_manager.cc:5649] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:33.489852 21847 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet fe5d794166cc4fbb87c6b918fa26b592 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:33.498368 22096 tablet_service.cc:1552] Processing DeleteTablet for tablet fe5d794166cc4fbb87c6b918fa26b592 with delete_type TABLET_DATA_TOMBSTONED (TS ca1362b9d9b34073bf35b6f6c9d1159a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39634
I20251023 02:04:33.500665 22879 tablet_replica.cc:333] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a: stopping tablet replica
I20251023 02:04:33.501756 22879 raft_consensus.cc:2243] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:33.502535 22879 raft_consensus.cc:2272] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:33.505993 22881 tablet_replica.cc:333] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14: stopping tablet replica
I20251023 02:04:33.506984 22881 raft_consensus.cc:2243] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:33.507727 22881 raft_consensus.cc:2272] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:33.510320 22880 raft_consensus.cc:2243] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:33.511128 22880 raft_consensus.cc:2272] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:33.511487 22879 ts_tablet_manager.cc:1916] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:33.515966 22880 ts_tablet_manager.cc:1916] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:33.519990 22881 ts_tablet_manager.cc:1916] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:33.534664 22880 ts_tablet_manager.cc:1929] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:33.535056 22880 log.cc:1199] T ffe0e348fc464e36be33082ddcdba159 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-1-root/wals/ffe0e348fc464e36be33082ddcdba159
I20251023 02:04:33.536341 21847 catalog_manager.cc:4985] TS 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): tablet ffe0e348fc464e36be33082ddcdba159 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:33.538642 22879 ts_tablet_manager.cc:1929] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:33.539093 22879 log.cc:1199] T fe5d794166cc4fbb87c6b918fa26b592 P ca1362b9d9b34073bf35b6f6c9d1159a: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-2-root/wals/fe5d794166cc4fbb87c6b918fa26b592
I20251023 02:04:33.541004 21844 catalog_manager.cc:4985] TS ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): tablet fe5d794166cc4fbb87c6b918fa26b592 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:33.548746 22881 ts_tablet_manager.cc:1929] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:33.549249 22881 log.cc:1199] T c0a3abec901e4c2eba45233d075d9263 P 02c53ba06cc3445a8534a84aa76b1c14: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-0-root/wals/c0a3abec901e4c2eba45233d075d9263
I20251023 02:04:33.550464 21846 catalog_manager.cc:4985] TS 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): tablet c0a3abec901e4c2eba45233d075d9263 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:34.566553 22109 consensus_queue.cc:237] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:34.572033 22033 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251023 02:04:34.572568 22075 consensus_peers.cc:597] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf578f7b0cdb4188bb43d676b1c64e28. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:34.573360 21958 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:34.573556 22851 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:34.575659 22856 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:34.583060 22856 raft_consensus.cc:2955] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.584474 22035 raft_consensus.cc:2955] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.588541 21958 raft_consensus.cc:2955] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.598094 21960 consensus_queue.cc:237] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:34.606492 21857 catalog_manager.cc:5649] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:34.611270 22109 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:34.612856 22861 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:34.613662 22035 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:34.614907 22861 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251023 02:04:34.616279 21926 consensus_peers.cc:597] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2b6e2b19ddc453290a76212731a2f28. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:34.622176 22846 raft_consensus.cc:2955] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.628445 22109 raft_consensus.cc:2955] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.629356 22035 raft_consensus.cc:2955] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.685554 22033 consensus_queue.cc:237] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
W20251023 02:04:34.693908 22001 consensus_peers.cc:597] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66c8e1c6015d47879ab7927548ad3b02. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:34.694979 22109 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:34.696559 22895 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:34.696825 21960 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:34.709072 21859 catalog_manager.cc:5649] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.715731 22894 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:34.715297 22704 raft_consensus.cc:2955] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.728531 22111 raft_consensus.cc:2955] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.734844 22109 consensus_queue.cc:237] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:34.735514 21960 raft_consensus.cc:2955] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.740576 21856 catalog_manager.cc:5649] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251023 02:04:34.753533 22075 consensus_peers.cc:597] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1aaf7b9acded449f9017f47455026853. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:34.754024 22035 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:34.754141 21960 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:34.755337 22883 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:34.757055 22856 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:34.782092 22856 raft_consensus.cc:2955] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.783584 22030 raft_consensus.cc:2955] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.784444 21958 raft_consensus.cc:2955] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:34.801658 21859 catalog_manager.cc:5649] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:35.003372 22904 ts_tablet_manager.cc:933] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499)
I20251023 02:04:35.005023 22904 tablet_copy_client.cc:323] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.1:41499
I20251023 02:04:35.006248 21971 tablet_copy_service.cc:140] P 02c53ba06cc3445a8534a84aa76b1c14: Received BeginTabletCopySession request for tablet d2b6e2b19ddc453290a76212731a2f28 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:36488)
I20251023 02:04:35.006736 21971 tablet_copy_service.cc:161] P 02c53ba06cc3445a8534a84aa76b1c14: Beginning new tablet copy session on tablet d2b6e2b19ddc453290a76212731a2f28 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:36488: session id = 1ab8b0b58147446d84d901b4ed2ff39e-d2b6e2b19ddc453290a76212731a2f28
I20251023 02:04:35.022804 21971 tablet_copy_source_session.cc:215] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:35.025890 22904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b6e2b19ddc453290a76212731a2f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:35.041661 22904 tablet_copy_client.cc:806] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:35.042124 22904 tablet_copy_client.cc:670] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:35.047219 22904 tablet_copy_client.cc:538] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:35.054013 22904 tablet_bootstrap.cc:492] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:35.075982 22904 tablet_bootstrap.cc:492] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:35.076997 22904 tablet_bootstrap.cc:492] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:35.077718 22904 ts_tablet_manager.cc:1403] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.004s
I20251023 02:04:35.080120 22904 raft_consensus.cc:359] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:35.080886 22904 raft_consensus.cc:740] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:35.082381 22904 consensus_queue.cc:260] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:35.092501 22904 ts_tablet_manager.cc:1434] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s
I20251023 02:04:35.094422 21971 tablet_copy_service.cc:342] P 02c53ba06cc3445a8534a84aa76b1c14: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-d2b6e2b19ddc453290a76212731a2f28 received from {username='slave'} at 127.0.0.1:36488
I20251023 02:04:35.094832 21971 tablet_copy_service.cc:434] P 02c53ba06cc3445a8534a84aa76b1c14: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-d2b6e2b19ddc453290a76212731a2f28 on tablet d2b6e2b19ddc453290a76212731a2f28 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:35.125416 22904 ts_tablet_manager.cc:933] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:35.128516 22904 tablet_copy_client.cc:323] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.3:33899
I20251023 02:04:35.129801 22121 tablet_copy_service.cc:140] P ca1362b9d9b34073bf35b6f6c9d1159a: Received BeginTabletCopySession request for tablet cf578f7b0cdb4188bb43d676b1c64e28 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:56894)
I20251023 02:04:35.130302 22121 tablet_copy_service.cc:161] P ca1362b9d9b34073bf35b6f6c9d1159a: Beginning new tablet copy session on tablet cf578f7b0cdb4188bb43d676b1c64e28 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:56894: session id = 1ab8b0b58147446d84d901b4ed2ff39e-cf578f7b0cdb4188bb43d676b1c64e28
I20251023 02:04:35.136909 22121 tablet_copy_source_session.cc:215] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:35.139571 22904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf578f7b0cdb4188bb43d676b1c64e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:35.153506 22904 tablet_copy_client.cc:806] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:35.154074 22904 tablet_copy_client.cc:670] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:35.157224 22904 tablet_copy_client.cc:538] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:35.164799 22904 tablet_bootstrap.cc:492] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:35.182139 22904 tablet_bootstrap.cc:492] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:35.182813 22904 tablet_bootstrap.cc:492] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:35.183342 22904 ts_tablet_manager.cc:1403] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.005s
I20251023 02:04:35.185735 22904 raft_consensus.cc:359] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:35.186291 22904 raft_consensus.cc:740] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:35.186831 22904 consensus_queue.cc:260] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:35.188745 22904 ts_tablet_manager.cc:1434] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251023 02:04:35.190474 22121 tablet_copy_service.cc:342] P ca1362b9d9b34073bf35b6f6c9d1159a: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-cf578f7b0cdb4188bb43d676b1c64e28 received from {username='slave'} at 127.0.0.1:56894
I20251023 02:04:35.190857 22121 tablet_copy_service.cc:434] P ca1362b9d9b34073bf35b6f6c9d1159a: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-cf578f7b0cdb4188bb43d676b1c64e28 on tablet cf578f7b0cdb4188bb43d676b1c64e28 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:35.254065 22904 ts_tablet_manager.cc:933] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:35.255450 22904 tablet_copy_client.cc:323] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.2:40645
I20251023 02:04:35.256685 22047 tablet_copy_service.cc:140] P 818907d63f1f4d84ba267ebe6ee837e0: Received BeginTabletCopySession request for tablet 66c8e1c6015d47879ab7927548ad3b02 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:55770)
I20251023 02:04:35.257215 22047 tablet_copy_service.cc:161] P 818907d63f1f4d84ba267ebe6ee837e0: Beginning new tablet copy session on tablet 66c8e1c6015d47879ab7927548ad3b02 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:55770: session id = 1ab8b0b58147446d84d901b4ed2ff39e-66c8e1c6015d47879ab7927548ad3b02
I20251023 02:04:35.271600 22047 tablet_copy_source_session.cc:215] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:35.274600 22904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c8e1c6015d47879ab7927548ad3b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:35.289497 22904 tablet_copy_client.cc:806] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:35.290021 22904 tablet_copy_client.cc:670] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:35.294327 22904 tablet_copy_client.cc:538] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:35.301355 22904 tablet_bootstrap.cc:492] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:35.322321 22904 tablet_bootstrap.cc:492] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:35.323663 22904 tablet_bootstrap.cc:492] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:35.324713 22904 ts_tablet_manager.cc:1403] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20251023 02:04:35.326442 22912 ts_tablet_manager.cc:933] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:35.327131 22904 raft_consensus.cc:359] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:35.327817 22904 raft_consensus.cc:740] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:35.328296 22904 consensus_queue.cc:260] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:35.330235 22904 ts_tablet_manager.cc:1434] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20251023 02:04:35.334481 22047 tablet_copy_service.cc:342] P 818907d63f1f4d84ba267ebe6ee837e0: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-66c8e1c6015d47879ab7927548ad3b02 received from {username='slave'} at 127.0.0.1:55770
I20251023 02:04:35.335045 22047 tablet_copy_service.cc:434] P 818907d63f1f4d84ba267ebe6ee837e0: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-66c8e1c6015d47879ab7927548ad3b02 on tablet 66c8e1c6015d47879ab7927548ad3b02 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:35.340080 22912 tablet_copy_client.cc:323] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.3:33899
I20251023 02:04:35.343066 22121 tablet_copy_service.cc:140] P ca1362b9d9b34073bf35b6f6c9d1159a: Received BeginTabletCopySession request for tablet 1aaf7b9acded449f9017f47455026853 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:56894)
I20251023 02:04:35.343590 22121 tablet_copy_service.cc:161] P ca1362b9d9b34073bf35b6f6c9d1159a: Beginning new tablet copy session on tablet 1aaf7b9acded449f9017f47455026853 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:56894: session id = 1ab8b0b58147446d84d901b4ed2ff39e-1aaf7b9acded449f9017f47455026853
I20251023 02:04:35.354239 22121 tablet_copy_source_session.cc:215] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:35.358707 22912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aaf7b9acded449f9017f47455026853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:35.371902 22912 tablet_copy_client.cc:806] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:35.372548 22912 tablet_copy_client.cc:670] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:35.377279 22912 tablet_copy_client.cc:538] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:35.384968 22912 tablet_bootstrap.cc:492] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:35.405006 22912 tablet_bootstrap.cc:492] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:35.405939 22912 tablet_bootstrap.cc:492] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:35.406469 22912 ts_tablet_manager.cc:1403] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.007s
I20251023 02:04:35.409122 22912 raft_consensus.cc:359] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:35.409953 22912 raft_consensus.cc:740] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:35.410605 22912 consensus_queue.cc:260] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:35.412956 22912 ts_tablet_manager.cc:1434] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20251023 02:04:35.414523 22121 tablet_copy_service.cc:342] P ca1362b9d9b34073bf35b6f6c9d1159a: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-1aaf7b9acded449f9017f47455026853 received from {username='slave'} at 127.0.0.1:56894
I20251023 02:04:35.414899 22121 tablet_copy_service.cc:434] P ca1362b9d9b34073bf35b6f6c9d1159a: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-1aaf7b9acded449f9017f47455026853 on tablet 1aaf7b9acded449f9017f47455026853 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:35.594758 22622 raft_consensus.cc:1217] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:35.627606 22622 raft_consensus.cc:1217] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:35.774128 22622 raft_consensus.cc:1217] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:35.815702 22622 raft_consensus.cc:1217] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:36.128260 22846 raft_consensus.cc:1064] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:36.130299 22846 consensus_queue.cc:237] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:36.135903 22107 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.135915 22622 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.138427 22846 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:36.139205 22897 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:36.139840 22030 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.142740 22914 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:36.163321 22919 raft_consensus.cc:2955] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.165457 22030 raft_consensus.cc:2955] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.176759 22622 raft_consensus.cc:2955] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.194270 21856 catalog_manager.cc:5649] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:04:36.174657 22109 raft_consensus.cc:2955] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.221773 22856 raft_consensus.cc:1064] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:36.224968 22856 consensus_queue.cc:237] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:36.232277 22622 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.234146 22915 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:36.235833 22030 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.238132 22856 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:36.238302 21960 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.239681 22856 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:36.246863 22878 raft_consensus.cc:1064] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:36.249768 22878 consensus_queue.cc:237] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:36.256098 22856 raft_consensus.cc:2955] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.260228 22030 raft_consensus.cc:2955] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.268661 21958 consensus_queue.cc:237] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:36.280421 22622 raft_consensus.cc:2955] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.290421 22624 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:36.293443 22897 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:04:36.296170 22030 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:36.297959 22111 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.305840 22897 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:36.306442 22878 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:36.307524 22914 raft_consensus.cc:2955] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.310580 21958 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.299523 22622 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.311494 22621 raft_consensus.cc:2955] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.312942 22878 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:36.314503 22704 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251023 02:04:36.316547 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 56106.8us
I20251023 02:04:36.322609 21846 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet d2b6e2b19ddc453290a76212731a2f28 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:36.323632 22030 raft_consensus.cc:2955] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.298830 21856 catalog_manager.cc:5649] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.345232 21857 catalog_manager.cc:5649] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 3 to 4, VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) evicted. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.353840 22878 raft_consensus.cc:2955] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.360105 22096 tablet_service.cc:1552] Processing DeleteTablet for tablet d2b6e2b19ddc453290a76212731a2f28 with delete_type TABLET_DATA_TOMBSTONED (TS ca1362b9d9b34073bf35b6f6c9d1159a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39634
I20251023 02:04:36.364681 21960 raft_consensus.cc:2955] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.365149 22622 raft_consensus.cc:2955] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.372817 22107 raft_consensus.cc:2955] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.392957 22915 raft_consensus.cc:1064] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:36.394575 21857 catalog_manager.cc:5649] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.395323 22915 consensus_queue.cc:237] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:36.400503 21957 raft_consensus.cc:2955] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.419334 22111 consensus_queue.cc:237] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:36.424593 21957 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.432722 22915 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:36.433790 22035 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.441603 22622 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:36.443910 22621 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:36.449694 22938 tablet_replica.cc:333] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a: stopping tablet replica
I20251023 02:04:36.450670 22938 raft_consensus.cc:2243] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:36.451304 22938 raft_consensus.cc:2272] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:36.455837 21960 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:36.456828 22934 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:36.457023 22932 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:36.483449 22883 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:36.484107 22035 consensus_queue.cc:237] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:36.498157 22109 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:36.500586 22622 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:36.504071 22933 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:04:36.505645 22939 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:36.483006 22938 ts_tablet_manager.cc:1916] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:36.514377 22941 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:36.523566 22883 raft_consensus.cc:2955] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.530792 22878 raft_consensus.cc:2955] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.534027 22621 raft_consensus.cc:2955] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.555224 21957 raft_consensus.cc:2955] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.558244 22915 raft_consensus.cc:2955] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.561010 21960 raft_consensus.cc:2955] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.585477 22109 raft_consensus.cc:2955] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
W20251023 02:04:36.590087 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 52846.3us
I20251023 02:04:36.616509 22938 ts_tablet_manager.cc:1929] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:36.617031 22938 log.cc:1199] T d2b6e2b19ddc453290a76212731a2f28 P ca1362b9d9b34073bf35b6f6c9d1159a: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-2-root/wals/d2b6e2b19ddc453290a76212731a2f28
I20251023 02:04:36.622834 21844 catalog_manager.cc:4985] TS ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): tablet d2b6e2b19ddc453290a76212731a2f28 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:36.621707 22621 raft_consensus.cc:2955] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.625718 21857 catalog_manager.cc:5649] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.630340 21847 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 66c8e1c6015d47879ab7927548ad3b02 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:36.630862 22036 raft_consensus.cc:2955] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
W20251023 02:04:36.636364 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 44761.1us
I20251023 02:04:36.634696 21859 catalog_manager.cc:5649] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New cstate: current_term: 1 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:36.650476 22622 raft_consensus.cc:2955] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.651516 21948 tablet_service.cc:1552] Processing DeleteTablet for tablet 66c8e1c6015d47879ab7927548ad3b02 with delete_type TABLET_DATA_TOMBSTONED (TS 02c53ba06cc3445a8534a84aa76b1c14 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53732
I20251023 02:04:36.656148 21844 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet cf578f7b0cdb4188bb43d676b1c64e28 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:36.666489 22956 tablet_replica.cc:333] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14: stopping tablet replica
I20251023 02:04:36.667559 22956 raft_consensus.cc:2243] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:36.668547 22956 raft_consensus.cc:2272] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:36.672381 22020 tablet_service.cc:1552] Processing DeleteTablet for tablet cf578f7b0cdb4188bb43d676b1c64e28 with delete_type TABLET_DATA_TOMBSTONED (TS 818907d63f1f4d84ba267ebe6ee837e0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38996
I20251023 02:04:36.683596 22956 ts_tablet_manager.cc:1916] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:36.684969 22957 tablet_replica.cc:333] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0: stopping tablet replica
I20251023 02:04:36.681465 21856 catalog_manager.cc:5649] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.687451 22957 raft_consensus.cc:2243] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:36.688261 22957 raft_consensus.cc:2272] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:36.691952 22957 ts_tablet_manager.cc:1916] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:36.699573 22956 ts_tablet_manager.cc:1929] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:36.701822 22956 log.cc:1199] T 66c8e1c6015d47879ab7927548ad3b02 P 02c53ba06cc3445a8534a84aa76b1c14: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-0-root/wals/66c8e1c6015d47879ab7927548ad3b02
I20251023 02:04:36.703464 21846 catalog_manager.cc:4985] TS 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): tablet 66c8e1c6015d47879ab7927548ad3b02 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:36.708355 22957 ts_tablet_manager.cc:1929] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:36.708875 22957 log.cc:1199] T cf578f7b0cdb4188bb43d676b1c64e28 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-1-root/wals/cf578f7b0cdb4188bb43d676b1c64e28
I20251023 02:04:36.710688 21847 catalog_manager.cc:4985] TS 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): tablet cf578f7b0cdb4188bb43d676b1c64e28 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:36.714912 22107 consensus_queue.cc:237] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:36.722010 22621 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:36.722353 22035 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:36.723304 22955 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:36.724160 22915 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:36.729825 22883 raft_consensus.cc:2955] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.730665 22035 raft_consensus.cc:2955] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.733079 22621 raft_consensus.cc:2955] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.742233 21844 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 1aaf7b9acded449f9017f47455026853 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:36.747567 21856 catalog_manager.cc:5649] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index 3 to 4, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:36.762300 21948 tablet_service.cc:1552] Processing DeleteTablet for tablet 1aaf7b9acded449f9017f47455026853 with delete_type TABLET_DATA_TOMBSTONED (TS 02c53ba06cc3445a8534a84aa76b1c14 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53732
I20251023 02:04:36.763562 22956 tablet_replica.cc:333] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14: stopping tablet replica
I20251023 02:04:36.764595 22956 raft_consensus.cc:2243] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 02:04:36.765275 22956 raft_consensus.cc:2272] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:36.769971 22956 ts_tablet_manager.cc:1916] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:36.784123 22956 ts_tablet_manager.cc:1929] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:36.784574 22956 log.cc:1199] T 1aaf7b9acded449f9017f47455026853 P 02c53ba06cc3445a8534a84aa76b1c14: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-0-root/wals/1aaf7b9acded449f9017f47455026853
I20251023 02:04:36.785887 21846 catalog_manager.cc:4985] TS 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): tablet 1aaf7b9acded449f9017f47455026853 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:37.815558 21960 consensus_queue.cc:237] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
W20251023 02:04:37.820607 21926 consensus_peers.cc:597] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 422ac2af786e439185d0e762e06ef0a3. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:37.820587 22035 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:37.820580 22109 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:37.822060 22919 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:37.824061 22919 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:37.829317 22846 raft_consensus.cc:2955] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:37.830587 22036 raft_consensus.cc:2955] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:37.842227 21960 consensus_queue.cc:237] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:37.851760 22109 raft_consensus.cc:2955] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:37.853401 22107 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:37.853621 22036 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251023 02:04:37.854300 21926 consensus_peers.cc:597] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 864751dbbbb748769206d89ccd8cb70e. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:37.845366 21856 catalog_manager.cc:5649] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:37.886869 22959 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:37.888351 22846 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:37.900815 22959 raft_consensus.cc:2955] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:37.909720 22036 raft_consensus.cc:2955] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:37.923384 22109 consensus_queue.cc:237] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:37.929028 22107 raft_consensus.cc:2955] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
W20251023 02:04:37.931802 22075 consensus_peers.cc:597] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dddaeefc3d449a0ba9b3e6695e37507. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:37.931998 22036 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:37.932837 21857 catalog_manager.cc:5649] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:37.934487 21958 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:37.934315 22958 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:37.936424 22954 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:37.957456 22954 raft_consensus.cc:2955] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:37.963613 22036 raft_consensus.cc:2955] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:37.970016 21958 raft_consensus.cc:2955] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:37.971433 22107 consensus_queue.cc:237] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:37.981317 21858 catalog_manager.cc:5649] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
W20251023 02:04:37.987694 22075 consensus_peers.cc:597] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a -> Peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Couldn't send request to peer 1ab8b0b58147446d84d901b4ed2ff39e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e923b66ccf04dd8985bbfe31adbf0b4. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:37.989081 21960 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:37.994220 22035 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 02:04:37.995262 22954 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:37.996593 22973 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:38.004909 22954 raft_consensus.cc:2955] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:38.006213 21960 raft_consensus.cc:2955] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:38.007639 22035 raft_consensus.cc:2955] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:38.027766 21857 catalog_manager.cc:5649] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) added. New cstate: current_term: 1 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
I20251023 02:04:38.281380 22978 ts_tablet_manager.cc:933] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499)
I20251023 02:04:38.283170 22978 tablet_copy_client.cc:323] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.1:41499
I20251023 02:04:38.284488 21971 tablet_copy_service.cc:140] P 02c53ba06cc3445a8534a84aa76b1c14: Received BeginTabletCopySession request for tablet 422ac2af786e439185d0e762e06ef0a3 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:36488)
I20251023 02:04:38.284906 21971 tablet_copy_service.cc:161] P 02c53ba06cc3445a8534a84aa76b1c14: Beginning new tablet copy session on tablet 422ac2af786e439185d0e762e06ef0a3 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:36488: session id = 1ab8b0b58147446d84d901b4ed2ff39e-422ac2af786e439185d0e762e06ef0a3
I20251023 02:04:38.291548 21971 tablet_copy_source_session.cc:215] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:38.295398 22978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422ac2af786e439185d0e762e06ef0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:38.310459 22978 tablet_copy_client.cc:806] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:38.311111 22978 tablet_copy_client.cc:670] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:38.315043 22978 tablet_copy_client.cc:538] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:38.321362 22978 tablet_bootstrap.cc:492] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:38.337132 22978 tablet_bootstrap.cc:492] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:38.337873 22978 tablet_bootstrap.cc:492] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:38.338364 22978 ts_tablet_manager.cc:1403] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.004s
I20251023 02:04:38.340504 22978 raft_consensus.cc:359] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:38.341199 22978 raft_consensus.cc:740] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:38.341701 22978 consensus_queue.cc:260] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:38.351419 22982 ts_tablet_manager.cc:933] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:38.352710 22978 ts_tablet_manager.cc:1434] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.014s user 0.003s sys 0.004s
I20251023 02:04:38.358702 21971 tablet_copy_service.cc:342] P 02c53ba06cc3445a8534a84aa76b1c14: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-422ac2af786e439185d0e762e06ef0a3 received from {username='slave'} at 127.0.0.1:36488
I20251023 02:04:38.359262 21971 tablet_copy_service.cc:434] P 02c53ba06cc3445a8534a84aa76b1c14: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-422ac2af786e439185d0e762e06ef0a3 on tablet 422ac2af786e439185d0e762e06ef0a3 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:38.362052 22982 tablet_copy_client.cc:323] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.3:33899
I20251023 02:04:38.363505 22121 tablet_copy_service.cc:140] P ca1362b9d9b34073bf35b6f6c9d1159a: Received BeginTabletCopySession request for tablet 6dddaeefc3d449a0ba9b3e6695e37507 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:56894)
I20251023 02:04:38.363992 22121 tablet_copy_service.cc:161] P ca1362b9d9b34073bf35b6f6c9d1159a: Beginning new tablet copy session on tablet 6dddaeefc3d449a0ba9b3e6695e37507 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:56894: session id = 1ab8b0b58147446d84d901b4ed2ff39e-6dddaeefc3d449a0ba9b3e6695e37507
I20251023 02:04:38.369443 22121 tablet_copy_source_session.cc:215] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:38.371973 22982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dddaeefc3d449a0ba9b3e6695e37507. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:38.389164 22982 tablet_copy_client.cc:806] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:38.389824 22982 tablet_copy_client.cc:670] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:38.395288 22982 tablet_copy_client.cc:538] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:38.404971 22982 tablet_bootstrap.cc:492] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:38.424476 22982 tablet_bootstrap.cc:492] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:38.425303 22982 tablet_bootstrap.cc:492] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:38.425856 22982 ts_tablet_manager.cc:1403] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.005s
I20251023 02:04:38.428153 22982 raft_consensus.cc:359] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:38.428761 22982 raft_consensus.cc:740] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:38.429486 22982 consensus_queue.cc:260] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:38.432723 22982 ts_tablet_manager.cc:1434] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20251023 02:04:38.434235 22121 tablet_copy_service.cc:342] P ca1362b9d9b34073bf35b6f6c9d1159a: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-6dddaeefc3d449a0ba9b3e6695e37507 received from {username='slave'} at 127.0.0.1:56894
I20251023 02:04:38.434623 22121 tablet_copy_service.cc:434] P ca1362b9d9b34073bf35b6f6c9d1159a: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-6dddaeefc3d449a0ba9b3e6695e37507 on tablet 6dddaeefc3d449a0ba9b3e6695e37507 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:38.475919 22982 ts_tablet_manager.cc:933] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499)
I20251023 02:04:38.477280 22982 tablet_copy_client.cc:323] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.1:41499
I20251023 02:04:38.478600 21971 tablet_copy_service.cc:140] P 02c53ba06cc3445a8534a84aa76b1c14: Received BeginTabletCopySession request for tablet 864751dbbbb748769206d89ccd8cb70e from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:36488)
I20251023 02:04:38.479012 21971 tablet_copy_service.cc:161] P 02c53ba06cc3445a8534a84aa76b1c14: Beginning new tablet copy session on tablet 864751dbbbb748769206d89ccd8cb70e from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:36488: session id = 1ab8b0b58147446d84d901b4ed2ff39e-864751dbbbb748769206d89ccd8cb70e
I20251023 02:04:38.484541 22978 ts_tablet_manager.cc:933] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Initiating tablet copy from peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:38.484889 21971 tablet_copy_source_session.cc:215] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:38.487145 22978 tablet_copy_client.cc:323] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.81.3:33899
I20251023 02:04:38.487705 22982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 864751dbbbb748769206d89ccd8cb70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:38.488582 22121 tablet_copy_service.cc:140] P ca1362b9d9b34073bf35b6f6c9d1159a: Received BeginTabletCopySession request for tablet 5e923b66ccf04dd8985bbfe31adbf0b4 from peer 1ab8b0b58147446d84d901b4ed2ff39e ({username='slave'} at 127.0.0.1:56894)
I20251023 02:04:38.489109 22121 tablet_copy_service.cc:161] P ca1362b9d9b34073bf35b6f6c9d1159a: Beginning new tablet copy session on tablet 5e923b66ccf04dd8985bbfe31adbf0b4 from peer 1ab8b0b58147446d84d901b4ed2ff39e at {username='slave'} at 127.0.0.1:56894: session id = 1ab8b0b58147446d84d901b4ed2ff39e-5e923b66ccf04dd8985bbfe31adbf0b4
I20251023 02:04:38.496168 22121 tablet_copy_source_session.cc:215] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 02:04:38.498929 22978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e923b66ccf04dd8985bbfe31adbf0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 02:04:38.509661 22982 tablet_copy_client.cc:806] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:38.510371 22982 tablet_copy_client.cc:670] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:38.513891 22982 tablet_copy_client.cc:538] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:38.514252 22978 tablet_copy_client.cc:806] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 0 data blocks...
I20251023 02:04:38.514853 22978 tablet_copy_client.cc:670] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Starting download of 1 WAL segments...
I20251023 02:04:38.518149 22978 tablet_copy_client.cc:538] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 02:04:38.524993 22982 tablet_bootstrap.cc:492] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:38.526724 22978 tablet_bootstrap.cc:492] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap starting.
I20251023 02:04:38.548309 22982 tablet_bootstrap.cc:492] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:38.549185 22982 tablet_bootstrap.cc:492] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:38.549865 22982 ts_tablet_manager.cc:1403] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.001s
I20251023 02:04:38.551913 22978 tablet_bootstrap.cc:492] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 02:04:38.552743 22978 tablet_bootstrap.cc:492] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Bootstrap complete.
I20251023 02:04:38.552616 22982 raft_consensus.cc:359] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:38.553476 22982 raft_consensus.cc:740] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:38.554549 22978 ts_tablet_manager.cc:1403] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.005s
I20251023 02:04:38.554138 22982 consensus_queue.cc:260] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:38.556645 22982 ts_tablet_manager.cc:1434] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251023 02:04:38.565192 21971 tablet_copy_service.cc:342] P 02c53ba06cc3445a8534a84aa76b1c14: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-864751dbbbb748769206d89ccd8cb70e received from {username='slave'} at 127.0.0.1:36488
I20251023 02:04:38.565634 21971 tablet_copy_service.cc:434] P 02c53ba06cc3445a8534a84aa76b1c14: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-864751dbbbb748769206d89ccd8cb70e on tablet 864751dbbbb748769206d89ccd8cb70e with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:38.571640 22978 raft_consensus.cc:359] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:38.572679 22978 raft_consensus.cc:740] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Initialized, Role: LEARNER
I20251023 02:04:38.573829 22978 consensus_queue.cc:260] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:38.578296 22978 ts_tablet_manager.cc:1434] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20251023 02:04:38.579802 22121 tablet_copy_service.cc:342] P ca1362b9d9b34073bf35b6f6c9d1159a: Request end of tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-5e923b66ccf04dd8985bbfe31adbf0b4 received from {username='slave'} at 127.0.0.1:56894
I20251023 02:04:38.580168 22121 tablet_copy_service.cc:434] P ca1362b9d9b34073bf35b6f6c9d1159a: ending tablet copy session 1ab8b0b58147446d84d901b4ed2ff39e-5e923b66ccf04dd8985bbfe31adbf0b4 on tablet 5e923b66ccf04dd8985bbfe31adbf0b4 with peer 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:38.734074 22622 raft_consensus.cc:1217] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:38.836436 22622 raft_consensus.cc:1217] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:38.902987 22622 raft_consensus.cc:1217] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:39.093693 22622 raft_consensus.cc:1217] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251023 02:04:39.322584 22919 raft_consensus.cc:1064] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:39.324656 22919 consensus_queue.cc:237] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:39.329548 22111 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:39.329681 22622 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:39.329854 22036 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:39.331046 22919 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:39.331995 22959 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:39.332696 22989 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:39.346730 22959 raft_consensus.cc:2955] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:39.350762 22111 raft_consensus.cc:2955] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:39.355530 22036 raft_consensus.cc:2955] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:39.371707 22622 raft_consensus.cc:2955] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
W20251023 02:04:39.600339 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 52841.1us
I20251023 02:04:40.184128 22998 raft_consensus.cc:1064] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
W20251023 02:04:40.186810 22622 consensus_meta.cc:287] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.563s user 0.007s sys 0.000s
I20251023 02:04:40.406638 22846 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:40.407505 22846 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:40.410372 22846 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:40.455255 22959 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:40.456115 22959 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:40.458609 22959 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:40.459244 22959 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:40.459841 22959 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:40.462133 22959 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:40.542183 22846 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:40.543007 22846 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:40.545877 22846 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:40.588487 22952 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:40.589299 22952 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:40.591821 22952 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:40.698119 22959 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:40.698988 22959 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:40.702033 22959 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:40.911109 22959 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:40.812368 22952 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:40.912077 22952 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:40.914657 22952 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:40.915300 22952 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:40.720562 22846 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:40.916078 22846 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:40.918223 22846 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:40.918736 22846 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:40.919287 22846 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:40.921306 22846 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:40.921878 22846 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.108016 22846 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.110596 22846 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:40.976729 22959 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.121841 22959 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.122563 22959 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:40.959406 22952 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:41.124723 22952 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.212641 22959 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.215490 22959 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.239811 22959 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.299075 22959 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.300362 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 906) took 1925 ms. Trace:
I20251023 02:04:39.368866 21858 catalog_manager.cc:5649] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:41.285048 23007 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.305313 23007 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:41.306420 22000 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.000s sys 0.006s
W20251023 02:04:41.306855 22000 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.553s user 0.000s sys 0.006s
I20251023 02:04:41.288511 23003 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.307533 23007 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.307489 23003 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.309486 22952 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.310395 22952 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.300815 21958 rpcz_store.cc:276] 1023 02:04:39.375193 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:41.299446 (+1924253us) service_pool.cc:225] Handling call
1023 02:04:41.299711 (+ 265us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:41.299886 (+ 175us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:41.300038 (+ 152us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:41.300092 (+ 54us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:41.300127 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:41.300190 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:41.300312 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:41.312652 22952 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:41.312786 23003 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:41.309962 23005 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.313841 23005 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.316192 23005 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:41.316705 23005 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.316774 22952 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.317250 23005 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.317327 22952 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.319108 22952 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.319100 23005 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.319896 23005 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.320389 23005 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:41.321394 22959 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.299723 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 915) took 1494 ms. Trace:
I20251023 02:04:41.322525 23005 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.322086 21956 rpcz_store.cc:276] 1023 02:04:39.804776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:41.298647 (+1493871us) service_pool.cc:225] Handling call
1023 02:04:41.299014 (+ 367us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:41.299216 (+ 202us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:41.299252 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:41.299308 (+ 56us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:41.299345 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:41.299408 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:41.299555 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:41.326012 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 912) took 1522 ms. Trace:
I20251023 02:04:41.326406 21957 rpcz_store.cc:276] 1023 02:04:39.803962 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:41.325277 (+1521315us) service_pool.cc:225] Handling call
1023 02:04:41.325555 (+ 278us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:41.325737 (+ 182us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:41.325761 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:41.325796 (+ 35us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:41.325821 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:41.325871 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:41.325965 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:41.415247 23005 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.416060 23005 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.422959 23005 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:41.436445 22622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.249s user 0.009s sys 0.023s
W20251023 02:04:41.436825 22622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.249s user 0.009s sys 0.023s
I20251023 02:04:41.437472 22622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 167) took 2068 ms. Trace:
I20251023 02:04:41.437837 22622 rpcz_store.cc:276] 1023 02:04:39.368932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:39.371173 (+ 2241us) service_pool.cc:225] Handling call
1023 02:04:39.371450 (+ 277us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:39.371616 (+ 166us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:04:40.187260 (+815644us) log.cc:844] Serialized 14 byte log entry
1023 02:04:41.437120 (+1249860us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:41.437165 (+ 45us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:04:41.437207 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:41.437289 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:41.437402 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1097439,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":138,"thread_start_us":1249595,"threads_started":1}
W20251023 02:04:41.441202 22076 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.952s user 0.014s sys 0.130s
W20251023 02:04:41.441488 22076 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.952s user 0.014s sys 0.130s
I20251023 02:04:41.429395 23012 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.431473 23014 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.442229 23012 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.442708 23014 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.443723 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 908) took 2030 ms. Trace:
I20251023 02:04:41.444075 21960 rpcz_store.cc:276] 1023 02:04:39.413377 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:41.443028 (+2029651us) service_pool.cc:225] Handling call
1023 02:04:41.443299 (+ 271us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:41.443440 (+ 141us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:41.443463 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:41.443501 (+ 38us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:41.443529 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:41.443579 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:41.443670 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:41.446920 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 911) took 1644 ms. Trace:
I20251023 02:04:41.432183 23013 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.447268 21962 rpcz_store.cc:276] 1023 02:04:39.802226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:41.445970 (+1643744us) service_pool.cc:225] Handling call
1023 02:04:41.446246 (+ 276us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:41.446431 (+ 185us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:41.446466 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:41.446519 (+ 53us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:41.446561 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:41.446628 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:41.446759 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:41.447962 23013 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.437179 23005 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.450359 23013 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.450230 23005 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.450970 23014 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.444465 23012 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:41.453451 23007 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.454070 23007 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:41.455860 23007 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.459972 23013 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.460280 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 916) took 1655 ms. Trace:
I20251023 02:04:41.460741 23013 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.463394 23014 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.463987 23014 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.466128 23012 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.466771 23012 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.467995 23017 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.471153 23017 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.468757 23014 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.460800 21954 rpcz_store.cc:276] 1023 02:04:39.805020 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:41.459197 (+1654177us) service_pool.cc:225] Handling call
1023 02:04:41.459586 (+ 389us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:41.459820 (+ 234us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:41.459861 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:41.459918 (+ 57us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:41.459979 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:41.460054 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:41.460189 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:41.468350 23013 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.474174 23013 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.475097 23013 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.474658 23014 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.478087 21858 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56872 (request call id 207) took 2112 ms. Trace:
I20251023 02:04:41.482710 23005 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.488229 23007 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.488942 23007 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.489759 23002 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.490096 22959 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.490310 23002 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.491987 23007 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:41.492486 22846 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.492936 23003 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.492976 22846 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.493759 23003 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:41.494663 22077 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.085s user 0.007s sys 0.033s
W20251023 02:04:41.495059 22077 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.086s user 0.007s sys 0.034s
W20251023 02:04:41.495330 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 2.08641e+06us
I20251023 02:04:41.496124 23003 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.496953 23004 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.497429 23004 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:41.493494 23002 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.499212 23002 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.499568 23004 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.500192 23004 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.500751 23004 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.499634 23002 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.505381 23003 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.507781 23019 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.508073 23003 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.490571 22959 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:41.514626 22075 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.050s user 0.030s sys 0.115s
I20251023 02:04:41.514914 22959 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.514966 23003 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.515496 22959 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.515991 23005 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.516047 22959 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.516744 23005 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:41.515028 22075 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.051s user 0.030s sys 0.115s
I20251023 02:04:41.502316 23007 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.518533 23007 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.477890 23014 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.520434 23007 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.521215 23014 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:41.521865 23014 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.522666 22959 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.479278 21858 rpcz_store.cc:276] 1023 02:04:39.365733 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:39.365887 (+ 154us) service_pool.cc:225] Handling call
1023 02:04:41.478007 (+2112120us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:04:41.324130 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:04:41.324691 (+ 561us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:04:41.324727 (+ 36us) write_op.cc:435] Acquired schema lock
1023 02:04:41.324742 (+ 15us) tablet.cc:660] Decoding operations
1023 02:04:41.325087 (+ 345us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:04:41.325145 (+ 58us) write_op.cc:641] Partition lock acquired for write op
1023 02:04:41.325176 (+ 31us) tablet.cc:683] Acquiring locks for 1 operations
1023 02:04:41.325342 (+ 166us) tablet.cc:699] Row locks acquired
1023 02:04:41.325358 (+ 16us) write_op.cc:260] PREPARE: finished
1023 02:04:41.325479 (+ 121us) write_op.cc:270] Start()
1023 02:04:41.325600 (+ 121us) write_op.cc:276] Timestamp: P: 1761185081325439 usec, L: 0
1023 02:04:41.325619 (+ 19us) op_driver.cc:348] REPLICATION: starting
1023 02:04:41.327711 (+ 2092us) log.cc:844] Serialized 590 byte log entry
1023 02:04:41.332992 (+ 5281us) op_driver.cc:464] REPLICATION: finished
1023 02:04:41.334127 (+ 1135us) write_op.cc:301] APPLY: starting
1023 02:04:41.334208 (+ 81us) tablet.cc:1365] starting BulkCheckPresence
1023 02:04:41.334306 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
1023 02:04:41.334401 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:04:41.334553 (+ 152us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:04:41.334661 (+ 108us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 02:04:41.334703 (+ 42us) write_op.cc:312] APPLY: finished
1023 02:04:41.334831 (+ 128us) log.cc:844] Serialized 22 byte log entry
1023 02:04:41.334976 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:41.335054 (+ 78us) write_op.cc:454] Released schema lock
1023 02:04:41.335088 (+ 34us) write_op.cc:341] FINISH: Updating metrics
1023 02:04:41.335366 (+ 278us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:41.335394 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":921,"apply.run_cpu_time_us":1243,"apply.run_wall_time_us":1243,"num_ops":1,"prepare.queue_time_us":17271,"prepare.run_cpu_time_us":3228,"prepare.run_wall_time_us":4701,"raft.queue_time_us":1431,"raft.run_cpu_time_us":1077,"raft.run_wall_time_us":2066,"replication_time_us":6344,"thread_start_us":19302,"threads_started":4,"wal-append.queue_time_us":994}]]}
I20251023 02:04:41.504546 22846 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:41.504680 22074 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.935s user 0.043s sys 0.085s
I20251023 02:04:41.515703 23003 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.508294 23019 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.510403 21857 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56864 (request call id 217) took 1905 ms. Trace:
W20251023 02:04:41.524360 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 2.06074e+06us
W20251023 02:04:41.524674 22074 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.955s user 0.043s sys 0.085s
I20251023 02:04:41.511519 23021 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.512233 23023 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.505607 23006 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.512051 21954 consensus_queue.cc:237] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.503738 23004 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.524852 21857 rpcz_store.cc:276] 1023 02:04:39.808201 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:41.390518 (+1582317us) service_pool.cc:225] Handling call
1023 02:04:41.510350 (+119832us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31993,"reported_tablets":1}
I20251023 02:04:41.523290 22998 consensus_queue.cc:237] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.522397 23014 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:41.530303 23006 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.537956 22954 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.541432 22954 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.543457 22954 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.543985 22954 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.544457 22954 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.523563 23003 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.428354 22952 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.529395 22958 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.525038 23005 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.568933 22954 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:41.569449 22954 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.569947 22954 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.571591 22954 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.560811 22952 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:41.572852 23027 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.573930 22952 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.573558 23027 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.574528 22952 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.574996 22952 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.576310 23027 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.576612 22952 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.578271 23030 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.582177 23020 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.583205 23029 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.582830 23020 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.475996 23012 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.478852 23016 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.581202 23005 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.576992 23027 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.591409 23025 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:04:41.593182 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 66749.9us
I20251023 02:04:41.597561 23019 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.534162 23021 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.605311 22952 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.477402 23013 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:41.605731 22952 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.590142 23017 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.568243 23023 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.593070 23016 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.616583 23023 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.616909 23016 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:41.617213 23023 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.588034 23030 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.583932 22958 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.598755 23005 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.610244 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.613407 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.605913 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.605914 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.719252 22952 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.719417 23023 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.719235 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.720243 23019 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.720602 23015 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.624964 23017 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.720794 23019 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.599476 23012 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.583865 23029 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.722620 21960 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:41.723169 21957 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:41.663494 22954 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.725270 22954 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.721176 23015 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.740903 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.743146 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.743634 21959 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:41.612771 23013 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.638756 23018 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.721504 23017 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.719235 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.604441 23027 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.730194 23012 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.861832 23013 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.744297 22954 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.864539 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.777935 23003 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.639076 23033 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.746167 23014 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.742249 23006 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.741796 23019 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.861086 23030 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.630923 23025 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.877749 23027 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.877902 23013 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.870301 21957 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:41.788694 23002 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.852658 23004 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.797365 23021 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.826463 23032 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.831035 23038 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.882474 23015 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.920203 23029 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.895047 23025 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.901593 23012 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:41.862121 23018 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.909134 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.919945 23032 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.911773 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.912189 22954 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.925591 22954 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.912390 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.912557 23034 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.913213 23036 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.913900 23051 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.916790 23050 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.918778 23042 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.899757 23027 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.892553 23033 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.923545 21959 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:41.922327 23002 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.892843 23030 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.928354 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.853229 23043 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.885963 23035 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.924142 23025 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.929611 23045 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.919860 23020 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.931946 22954 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.869728 23024 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:41.881711 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.910817 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.924034 23005 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:41.937915 21958 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:41.947575 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.924793 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.939796 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.949680 22954 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.921039 23004 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.951697 22954 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.954388 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.955199 21957 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:41.943982 23035 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.936990 23027 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.932005 23050 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.963603 23033 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.964938 23035 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.938378 23034 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.967862 23034 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.969199 23050 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.970162 23046 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.971719 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.972308 22954 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.974321 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.976651 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.943096 22958 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.931998 23040 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.944072 23024 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.981894 23049 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.984090 23033 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.978583 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.924034 23038 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.987493 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.970665 23046 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.956707 23020 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.942714 23025 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:41.994710 23033 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.979028 21955 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:41.995998 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:42.075695 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:41.998751 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:42.077931 21928 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.588s (SENT) [suppressed 1 similar messages]
I20251023 02:04:42.078308 21961 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
W20251023 02:04:42.076517 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:42.032963 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.082367 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.081504 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.040146 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.002399 23024 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.045130 23025 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.038403 23032 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.099478 22623 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.002332 23033 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.938907 23002 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.001659 23035 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.102057 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.939527 23036 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.997009 23040 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.102069 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:41.932142 23042 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.032127 23044 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.001266 22958 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.956632 23045 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.050638 23046 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.110991 23044 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.994272 23027 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:41.998508 23049 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:41.940024 23030 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.051798 23038 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.117378 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.118517 21955 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:41.994922 23056 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:41.946323 23043 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:41.998452 23004 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.040782 23023 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.032836 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:41.940979 23051 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.002358 22954 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:41.999888 23020 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.037672 23018 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:42.095219 23054 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.040870 22589 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
I20251023 02:04:42.101723 23035 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.102236 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.105821 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.101311 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.096447 23058 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.111344 22958 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:42.104565 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.111419 23040 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.097290 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.121098 23030 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:42.153072 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 75410.7us
I20251023 02:04:42.189715 23035 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.121701 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.117414 23045 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.152241 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.164253 23044 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.182767 23055 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.201155 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.164535 23042 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.156342 23049 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.206703 23025 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.206692 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.208638 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.209131 23020 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:42.197173 23058 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.209861 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.183274 23036 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.212386 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.213148 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.214877 23057 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.214814 23025 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.215272 22002 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:42.190528 23054 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.215832 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.217665 23002 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.215752 23057 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.213961 22623 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.213323 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.197307 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.221071 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.226408 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.233217 22002 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:42.219537 23035 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.223657 21960 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.213330 22998 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.230858 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.222015 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.221179 22958 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.234922 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.241505 23047 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.240433 22998 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.244202 23058 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.244027 22954 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.127409 23056 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.246412 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.248099 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.245003 23047 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:42.249255 23047 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.234917 23035 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.251618 23002 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.253002 23002 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:42.253432 23057 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:42.256073 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.258307 22618 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.245160 23025 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.263295 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.253433 23002 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.265913 22587 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:42.263216 22103 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.245600 23032 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:04:42.268780 23039 log.cc:927] Time spent T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.362s user 0.004s sys 0.000s
I20251023 02:04:42.269356 23044 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.252543 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.245556 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:42.251789 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 257554us
I20251023 02:04:42.273471 23033 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.279145 22998 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.272776 21956 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:42.283140 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.290907 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.291242 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.291288 23032 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.291966 22105 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.292981 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.284193 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.284435 22623 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.284739 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.261899 22619 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.217721 23055 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.296010 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.296859 23033 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.294320 22954 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:42.297863 23041 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.380s user 0.001s sys 0.002s
I20251023 02:04:42.309829 23017 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.311178 23054 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.292996 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.299868 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.293888 23032 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:42.303534 22002 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:42.312036 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.316398 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.296211 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.313839 23035 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:42.293980 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.306591 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.312286 23033 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.317991 22104 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.319551 23017 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:42.319478 22103 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.319906 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.324122 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.326925 23056 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.326987 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.332674 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.327256 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.325796 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.329232 23017 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:42.330260 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.330332 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.333595 23017 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:42.330857 23055 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.331055 22109 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:42.333539 22110 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:42.338321 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.328852 23023 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.350630 21962 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:42.351606 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.352391 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.352088 22109 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:42.348908 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.334985 22103 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.352173 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.358027 22110 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.358516 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.358667 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.359017 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.357682 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.356248 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.376204 22002 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:42.359823 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.359727 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.403505 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.362306 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.366298 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.422096 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.359608 22110 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:42.357096 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.425071 22111 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:42.425686 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.383262 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.350848 23023 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
W20251023 02:04:42.429601 22645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d71c0 after lost signal to thread 20807
I20251023 02:04:42.433017 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:42.433511 22645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d78a0 after lost signal to thread 21894
I20251023 02:04:42.389225 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.437175 22104 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
W20251023 02:04:42.439030 22645 debug-util.cc:398] Leaking SignalData structure 0x7b080020d200 after lost signal to thread 21986
I20251023 02:04:42.440536 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:42.440869 22645 debug-util.cc:398] Leaking SignalData structure 0x7b08003046c0 after lost signal to thread 22061
I20251023 02:04:42.441293 22105 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
W20251023 02:04:42.442421 22645 debug-util.cc:398] Leaking SignalData structure 0x7b08002e4020 after lost signal to thread 22135
I20251023 02:04:42.442618 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.434509 23023 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.444044 22618 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20251023 02:04:42.444183 22645 debug-util.cc:398] Leaking SignalData structure 0x7b08001c8a80 after lost signal to thread 22648
I20251023 02:04:42.445601 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:42.446089 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 108220us
I20251023 02:04:42.357280 22102 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.452581 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.453967 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.452503 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.454846 22033 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.456030 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.456171 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.452828 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.457451 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.454360 22111 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.458099 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.459275 22036 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:42.459461 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.459638 23016 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.461382 23016 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:42.461858 23016 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.464454 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.465502 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.465979 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.467056 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.467597 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.467820 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.468225 22109 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.472868 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.475220 23051 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.475317 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.475237 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.478320 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.460192 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.478875 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.479802 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.481420 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.481818 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.483000 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.483240 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.485090 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.485231 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.485865 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.483923 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:42.484377 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:42.480458 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.492049 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.492937 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.491041 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.493840 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.493733 22106 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.495922 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.496054 23017 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.492434 22110 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.497648 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.497742 22076 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:42.498399 22109 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.498556 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.498977 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.503547 22106 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.504074 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.504766 22107 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.501552 22110 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.502140 22958 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.505662 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.503522 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.506557 22109 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:42.506853 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.506778 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.505703 22958 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.508277 22958 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.499393 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.507216 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.484994 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.503672 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.510735 23043 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.501013 23002 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.514667 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.516363 23002 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.518375 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.519464 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.519783 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.520331 22032 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:42.520524 22036 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:42.522536 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.524315 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.525020 22028 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:42.520860 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.526466 22031 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:42.527572 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.529894 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:42.531960 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.530519 22110 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.498301 23017 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.530754 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.533679 22002 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:04:42.535799 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:42.536123 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:42.536262 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.536851 23017 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.537704 23017 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:42.537974 22105 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251023 02:04:42.538167 23017 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.540973 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.536767 22002 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:42.546929 23005 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.547361 23005 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:42.547752 23005 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.550137 23023 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:42.551326 22033 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:42.551505 23056 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:42.553668 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.554998 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.555356 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.556144 22028 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:42.556272 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.557164 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.557768 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.557988 22037 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:42.558427 22033 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:42.554877 22031 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:42.567063 23047 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.568276 22030 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:42.568445 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.558665 22958 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.558482 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.569875 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.570094 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.570777 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.571640 22958 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.571852 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.572539 22036 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:42.574165 23016 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.560204 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.561496 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.559393 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.562413 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.561996 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.562880 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.563894 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.576547 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.576740 22109 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:42.569896 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.577438 23016 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.577782 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.579140 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.579365 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.580781 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:42.564400 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.566847 22620 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:42.567238 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.574389 22587 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.582597 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.562330 23005 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.583966 22587 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:42.585007 22587 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.585973 23044 raft_consensus.cc:2804] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.586372 23044 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.587186 23044 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.587251 23026 raft_consensus.cc:1064] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:42.588016 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.590798 23026 consensus_queue.cc:237] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.594075 23043 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.594508 23043 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.594952 23043 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.600881 23019 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:04:42.605072 23032 raft_consensus.cc:1064] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:42.607625 22077 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:42.607774 22587 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:42.608662 23035 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.609560 22587 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:42.609829 23045 raft_consensus.cc:2749] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.610414 23030 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.610528 22587 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:42.610781 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.610829 23030 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.611374 23045 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.611974 23045 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.612347 23045 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.611248 21925 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:42.611552 23030 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.598063 23002 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.621233 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.616247 23002 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.621165 23019 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.623171 23002 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.623275 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:42.621091 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.631239 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:42.631438 22000 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.608253 23032 consensus_queue.cc:237] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.634171 23023 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.634490 23005 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.633237 22587 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:42.634771 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.639003 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.635241 22077 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:42.636217 23047 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.632684 22000 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.622076 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:42.641568 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.641842 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.640632 23040 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.641495 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:42.642009 22587 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:42.637496 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.643045 23035 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.648780 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.657169 23017 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.653450 23043 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.653071 23045 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.665894 23040 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.668213 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.669211 23047 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.669534 23047 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.669941 23047 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.670555 23045 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.674458 22077 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:42.671926 22000 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:42.677515 23043 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.677871 23044 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.660121 23023 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.678911 22958 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.680179 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.681772 22077 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.682852 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:42.683452 23004 raft_consensus.cc:2955] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:42.686956 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.681802 21925 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:42.689507 22075 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:42.691915 22621 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:42.681977 23023 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.691697 23005 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.695518 23035 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.687314 22615 raft_consensus.cc:2955] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:42.696900 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.697213 23019 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.697439 23003 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:42.698190 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.711083 23019 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.713919 23019 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.698313 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.699285 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.699738 22077 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:42.699286 21925 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:42.702103 23035 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:04:42.709919 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 93760.8us
I20251023 02:04:42.711170 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.712487 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.717178 23035 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.710391 22000 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.714890 22028 raft_consensus.cc:2393] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:42.701493 23023 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.719619 22075 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:42.722883 23055 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.722972 23056 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.717833 23051 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20251023 02:04:42.724877 22075 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:42.727155 23003 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.727576 23003 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.727943 23003 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.726424 22002 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:42.729513 23005 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.729945 23005 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.712126 22958 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.732506 22077 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:42.731941 22618 raft_consensus.cc:1240] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:42.734275 23016 raft_consensus.cc:2804] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.730297 23005 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.734732 23016 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:42.735047 22002 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:42.717937 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:42.735141 23016 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.715972 23030 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.726133 23051 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.737442 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.742662 23018 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.743295 22028 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:42.746281 22077 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:42.784644 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 45521.8us
I20251023 02:04:42.752192 23044 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.749528 23004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:53732 (request call id 928) took 1237 ms. Trace:
I20251023 02:04:42.790835 22029 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:42.751053 22618 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 LEARNER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:42.751708 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:42.760265 23017 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:42.791508 23063 log.cc:927] Time spent T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
W20251023 02:04:42.791942 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 45713.6us
I20251023 02:04:42.802577 22077 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.785347 22000 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:42.751350 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.746968 23060 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.785971 23018 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.804175 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:42.805259 21925 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:42.793227 22958 consensus_queue.cc:1059] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20251023 02:04:42.808115 23023 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.800220 23004 rpcz_store.cc:276] 1023 02:04:41.509672 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:41.509864 (+ 192us) service_pool.cc:225] Handling call
1023 02:04:41.541044 (+ 31180us) log.cc:844] Serialized 487 byte log entry
1023 02:04:42.204922 (+663878us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:04:42.347618 (+142696us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b1800316420. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9606ea 00007f9c65fd4a29 00007f9c5c820990 00007f9c5c9e84d4 00007f9c5c9e8445 00007f9c5f9f2a5b 00007f9c5fa777b3 00007f9c5fa9f2e2 00007f9c5fa9759a
1023 02:04:42.745774 (+398156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":423563,"raft.queue_time_us":528282,"raft.run_cpu_time_us":2052,"raft.run_wall_time_us":84510,"spinlock_wait_cycles":43824128,"thread_start_us":194571,"threads_started":1,"wal-append.queue_time_us":364073}
I20251023 02:04:42.822715 22077 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.822733 23058 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.823457 23058 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.823630 23033 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.823868 23058 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.823918 22077 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:42.823985 23033 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.824601 23033 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.825094 22077 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.752416 23047 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:42.827706 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 67038us
I20251023 02:04:42.815392 23060 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:42.840081 21925 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:42.841837 22077 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:04:42.843226 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 57960.5us
I20251023 02:04:42.828181 23056 raft_consensus.cc:2749] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.829491 23065 raft_consensus.cc:2749] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.812659 23051 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20251023 02:04:42.832706 23067 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.847607 22077 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:42.849027 22958 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.850860 22958 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.851229 22958 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.853919 23006 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.854315 23006 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.854708 23006 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.872766 21925 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:42.842868 23060 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.862043 23055 raft_consensus.cc:3055] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
W20251023 02:04:42.897914 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 52487.6us
I20251023 02:04:42.880247 23030 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.859143 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:42.856513 23018 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.902117 23018 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.902518 23018 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.902626 21846 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 422ac2af786e439185d0e762e06ef0a3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251023 02:04:42.893172 23055 raft_consensus.cc:740] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:42.871119 21857 catalog_manager.cc:5649] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New cstate: current_term: 1 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:42.868825 23023 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:42.903947 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 56460.9us
I20251023 02:04:42.857836 23032 raft_consensus.cc:2749] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:04:42.905555 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 55169.4us
I20251023 02:04:42.910319 22077 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:42.848224 23067 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.863085 23047 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.877511 23035 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:42.917035 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 44382.9us
I20251023 02:04:42.872051 23002 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.923031 23005 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.925029 23005 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.925770 23051 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.924993 23016 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:42.928180 23016 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.928573 22002 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:42.864255 23019 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.914425 23067 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:04:42.931026 21846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 50889.2us
I20251023 02:04:42.935101 22108 raft_consensus.cc:1240] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:42.935770 23019 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.932446 23065 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.932096 23071 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.937666 21925 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:42.928556 23055 consensus_queue.cc:260] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:42.942013 21928 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:42.936609 23065 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:42.938318 23071 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:42.947407 23065 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.948660 23071 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.953133 22077 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.953271 23026 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.956871 22077 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:42.958436 22846 raft_consensus.cc:2749] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.958662 23055 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.958923 23035 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.962677 22002 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:42.963006 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:42.966868 23072 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.967257 23072 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:42.967635 23072 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.957813 23032 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.965291 23002 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:42.970559 22077 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.976307 22000 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:42.969746 23035 consensus_queue.cc:1059] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:42.978996 23033 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.980576 22001 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.981443 23033 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:42.981957 23023 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.971295 23032 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.986332 23035 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:42.986745 23056 raft_consensus.cc:3055] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:42.974783 23006 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:42.992313 22000 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:42.990685 22077 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:42.994105 21928 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.000169 22001 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:42.998170 23056 raft_consensus.cc:740] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:42.999221 23017 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.001982 23068 raft_consensus.cc:2749] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:42.969096 23074 raft_consensus.cc:2749] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.002072 23017 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:42.993820 23032 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.005271 22001 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:42.994021 23035 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:43.005798 23073 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.006366 22620 raft_consensus.cc:1240] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.017369 22000 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:43.019143 21928 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.005615 23023 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.011233 23017 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:42.998054 23058 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.011759 23033 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.023411 23068 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.004413 23065 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.021080 23005 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.016933 23035 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:04:43.027940 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 95140.7us
I20251023 02:04:43.030628 22000 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:43.005776 23056 consensus_queue.cc:260] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.022783 23003 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.031956 21925 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:43.032126 23056 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.035616 23019 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.035974 23019 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:43.036315 23019 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.028613 23069 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.032884 21925 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:43.041218 22618 raft_consensus.cc:1240] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.039739 23069 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:43.045190 23069 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.027712 23033 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:43.028859 23005 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:43.046650 23003 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.047447 21928 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.046269 22001 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:43.038708 23067 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.047953 23005 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.051729 23073 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.052120 23073 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:43.052489 23073 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.053856 22000 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:43.030798 22958 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.057080 22846 consensus_queue.cc:1059] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:04:43.058672 22846 raft_consensus.cc:3055] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.058983 22846 raft_consensus.cc:740] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:43.047040 23033 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.060883 23004 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.061414 23067 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.061760 22846 consensus_queue.cc:260] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.063158 22846 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.064885 22001 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:43.084793 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:43.051756 23068 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.087097 23023 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.104960 23006 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.105012 23023 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:43.105677 23023 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.108428 21925 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:43.111124 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 146292us
W20251023 02:04:43.122076 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:43.148765 23081 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.150091 23081 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:43.150571 23081 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.100263 23068 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:43.154317 23068 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.109535 23076 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.053232 23026 consensus_queue.cc:1059] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
W20251023 02:04:43.192685 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 131562us
I20251023 02:04:43.183686 23076 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:43.195649 23076 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.195946 23067 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.196398 23067 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:43.196754 23067 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:04:43.197782 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 139040us
W20251023 02:04:43.197803 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 89462.8us
W20251023 02:04:43.199113 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 148335us
I20251023 02:04:43.200007 22000 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:43.205180 22001 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:43.237028 21925 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:43.249101 21925 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:43.234092 23047 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.233899 23065 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.234586 23058 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:43.278298 23060 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.201257 23018 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:43.280593 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 80728us
I20251023 02:04:43.286702 22000 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.258302 23047 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:43.297462 23047 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:04:43.297720 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 61658.8us
I20251023 02:04:43.296917 23026 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.296505 23070 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.306675 22958 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:43.314437 23084 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:04:43.320271 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 71264.3us
W20251023 02:04:43.314565 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 109478us
I20251023 02:04:43.328753 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:43.329099 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:43.331069 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:43.313226 23026 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:43.332954 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:43.317068 23070 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:43.335115 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:43.335062 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:43.335325 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.335474 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.336138 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.337270 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.337724 22590 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.340715 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:43.341681 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.342605 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:43.339380 23026 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.354638 21928 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.351984 21962 raft_consensus.cc:1240] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.357172 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:43.358362 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 71782.2us
I20251023 02:04:43.341238 23070 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.352195 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.364006 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.366122 23060 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.367239 23006 raft_consensus.cc:3055] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.367630 23006 raft_consensus.cc:740] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:43.368459 23006 consensus_queue.cc:260] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.373210 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:43.373011 23073 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.377063 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.370085 23067 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.350158 22618 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.343436 23086 raft_consensus.cc:2749] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:04:43.385195 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 47619us
I20251023 02:04:43.340997 23072 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.379264 23089 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.390192 22590 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.391335 23072 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.353051 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.384990 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.392541 22590 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.386175 23035 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.393621 23030 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.394037 23073 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:43.394217 23030 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.374172 23032 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.394616 23035 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:43.349119 23071 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.387001 21959 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:43.395771 21928 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.396142 23032 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:43.374593 23006 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.384065 23002 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.384024 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.385680 23019 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.398491 23089 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.382225 23018 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:43.383762 23076 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.399379 23019 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.380987 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:43.400813 23076 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.386777 23023 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:43.397513 21928 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.403961 23023 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.380437 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:43.405797 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:43.407922 23047 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:43.408989 23019 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.350792 23081 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.410307 21928 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.412547 21925 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:43.413296 23019 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.383093 23004 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.413982 23069 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.414844 23068 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.415819 23089 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.416365 23089 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:43.416780 23068 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.416833 23089 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.417495 21956 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.421357 21960 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:43.422224 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.423657 23023 consensus_queue.cc:1059] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:43.425333 23085 raft_consensus.cc:3055] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.425742 23085 raft_consensus.cc:740] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:43.425936 21925 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:43.426306 23017 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.426652 23085 consensus_queue.cc:260] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.427960 23085 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.384699 23005 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.431169 22000 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.432169 21959 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:43.433405 23005 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:43.436268 21928 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.428426 23017 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:43.438503 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:43.438295 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:43.438510 21928 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.437242 23087 raft_consensus.cc:2749] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:43.442471 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.431039 23082 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.442458 23023 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.444032 23082 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:43.445442 23082 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.445497 23003 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:04:43.449937 21959 raft_consensus.cc:1240] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.450243 23019 raft_consensus.cc:2749] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.451792 23071 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.452942 23067 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.428856 23026 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.460634 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.460958 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.456069 23068 consensus_queue.cc:1059] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20251023 02:04:43.461859 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.462136 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.462294 22103 raft_consensus.cc:1240] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.462767 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.463094 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:43.462976 23026 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.464496 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.466281 23069 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:43.466902 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 74503.4us
I20251023 02:04:43.469762 21959 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.470362 22000 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.471843 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.471951 22590 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.472460 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.457427 23081 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:43.477520 21959 raft_consensus.cc:1240] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.479292 22590 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.480006 23002 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.480764 23002 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.478104 22103 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.474025 23047 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.481757 22590 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.477813 23067 raft_consensus.cc:3055] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.482435 23067 raft_consensus.cc:740] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:43.483321 23002 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.483434 23047 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.484323 22590 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.484694 23067 consensus_queue.cc:260] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.481189 23033 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:43.485343 23002 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.483582 21928 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.486466 22590 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.487773 23002 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.488137 22590 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.489944 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.490053 23003 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.491632 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.491652 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:43.493232 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.494915 23002 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.494870 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.495424 21953 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20251023 02:04:43.495934 21961 raft_consensus.cc:2393] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:43.496459 22108 raft_consensus.cc:1240] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.496697 22590 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.499594 21928 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.494757 23067 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.501731 23002 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.502475 23033 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:43.505769 23006 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.485816 23070 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.507290 21957 raft_consensus.cc:1240] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.508260 23070 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.509553 21957 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.510689 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.488546 22110 raft_consensus.cc:2955] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:43.511217 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.513281 21928 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.514235 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.518085 21961 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:43.518290 23082 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.519663 23082 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.522209 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.527971 21960 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:43.529079 22000 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.530718 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.531009 21928 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.532802 23081 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.534401 22000 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.534809 22103 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:43.535377 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.535953 23082 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.536831 23069 raft_consensus.cc:2749] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.537176 23082 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:43.537940 23082 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.505151 23089 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.538434 23069 consensus_queue.cc:1059] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:43.539039 21961 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251023 02:04:43.539181 23071 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.539296 22000 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.539057 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.540480 23067 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:43.536823 23006 raft_consensus.cc:2749] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.546675 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.549167 23069 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.549866 23069 raft_consensus.cc:3055] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.550185 23069 raft_consensus.cc:740] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:43.542717 23085 consensus_queue.cc:1059] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20251023 02:04:43.550920 23069 consensus_queue.cc:260] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.552083 23069 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.552013 23085 raft_consensus.cc:3055] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.556638 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.557744 23047 consensus_queue.cc:1059] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:43.558995 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.559801 23047 raft_consensus.cc:3055] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.560144 23047 raft_consensus.cc:740] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:43.560508 21959 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:43.560988 23047 consensus_queue.cc:260] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.557219 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.558326 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.563628 23047 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.558651 22002 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:43.566013 23017 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:43.566533 23017 raft_consensus.cc:697] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:43.567466 23017 consensus_queue.cc:237] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.569712 21958 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.573206 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:43.573668 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:43.574348 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.574622 22002 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:43.575327 23081 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:43.575815 23081 raft_consensus.cc:697] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:43.577206 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.576707 23081 consensus_queue.cc:237] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:43.579072 22029 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.580574 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:43.581604 23085 raft_consensus.cc:740] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:43.582616 23085 consensus_queue.cc:260] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.583426 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.584053 22110 raft_consensus.cc:2393] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:43.587306 23023 consensus_queue.cc:1059] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:43.588052 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.588553 22036 raft_consensus.cc:1240] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.589326 23076 raft_consensus.cc:3055] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.589653 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.589814 23076 raft_consensus.cc:740] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:43.590436 22029 raft_consensus.cc:1240] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.590791 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.590896 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.590616 23076 consensus_queue.cc:260] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.597178 22589 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.584782 22103 raft_consensus.cc:1240] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.589329 23033 consensus_queue.cc:1059] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20251023 02:04:43.598425 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.598556 22589 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.599231 23033 raft_consensus.cc:3055] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:43.599587 23033 raft_consensus.cc:740] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:43.600520 23033 consensus_queue.cc:260] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.601420 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.597114 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.597729 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.589272 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.599536 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.597015 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.603932 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:43.604943 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 50299.5us
I20251023 02:04:43.602202 22035 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.608955 22077 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:43.611610 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.612943 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.615640 22030 raft_consensus.cc:1240] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.616295 22077 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.617419 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.604140 23002 raft_consensus.cc:2804] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:43.618880 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.619450 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.623996 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.624547 23035 raft_consensus.cc:2749] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.624783 22030 raft_consensus.cc:1240] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.602169 23018 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20251023 02:04:43.625195 23072 consensus_queue.cc:1059] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:43.625718 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.626204 23072 raft_consensus.cc:3055] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.626559 23072 raft_consensus.cc:740] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:43.626718 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.627408 23055 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:43.627808 23055 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:43.624117 23002 raft_consensus.cc:697] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:43.628204 23055 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.628815 23002 consensus_queue.cc:237] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.632504 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.628109 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.633561 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.627808 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.635172 22030 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.635277 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.635601 22036 raft_consensus.cc:1240] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.601747 23092 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:43.636200 22030 raft_consensus.cc:1240] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.636441 23092 raft_consensus.cc:697] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:43.636307 23082 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.636729 22036 raft_consensus.cc:1240] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.637259 22030 raft_consensus.cc:1240] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.637336 21957 raft_consensus.cc:1240] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.639115 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.640448 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.591343 23085 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.633834 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.644351 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.646292 21925 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.598124 23076 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.601558 23033 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:43.652597 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:43.653817 22035 raft_consensus.cc:1240] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.656083 22077 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:43.657402 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:43.657014 22036 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.657548 22077 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.661260 22033 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:43.661870 22077 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.663190 22030 raft_consensus.cc:1240] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.659772 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.603142 23032 consensus_queue.cc:1059] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20251023 02:04:43.664884 21928 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.652370 22037 raft_consensus.cc:1240] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.674202 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.655119 23073 consensus_queue.cc:1059] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251023 02:04:43.652889 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.679373 22108 raft_consensus.cc:2410] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20251023 02:04:43.683207 22077 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.692713 22621 raft_consensus.cc:1240] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.694723 22035 raft_consensus.cc:1240] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.681885 23047 raft_consensus.cc:3055] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.699823 22077 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.700671 23075 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.666564 23018 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.701052 23074 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.701283 22106 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.710754 22020 tablet_service.cc:1552] Processing DeleteTablet for tablet 422ac2af786e439185d0e762e06ef0a3 with delete_type TABLET_DATA_TOMBSTONED (TS 818907d63f1f4d84ba267ebe6ee837e0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38996
I20251023 02:04:43.701885 23073 consensus_queue.cc:1059] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:43.702028 23019 consensus_queue.cc:1059] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:43.627369 23072 consensus_queue.cc:260] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.703325 22002 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:43.705969 21928 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.708007 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.708478 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.708950 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.709508 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.709625 22000 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.709879 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.651958 23067 consensus_queue.cc:1059] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20251023 02:04:43.637362 23092 consensus_queue.cc:237] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.670979 23068 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.671368 23026 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.714512 22077 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.718678 23078 consensus_queue.cc:1059] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20251023 02:04:43.720953 23072 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.671236 23071 consensus_queue.cc:1059] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20251023 02:04:43.718751 22028 raft_consensus.cc:1240] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20251023 02:04:43.664672 23086 consensus_queue.cc:1059] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20251023 02:04:43.716320 23056 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.722744 23082 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.723213 23089 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:43.703935 23035 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.717507 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.742360 23086 raft_consensus.cc:3055] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.742717 23086 raft_consensus.cc:740] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:43.725641 23005 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.704535 23047 raft_consensus.cc:740] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:43.730711 21958 raft_consensus.cc:1240] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.732125 21957 raft_consensus.cc:1240] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.730427 22000 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.712090 21954 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:43.737017 22077 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.719190 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:43.749712 22000 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.750501 23023 raft_consensus.cc:2749] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251023 02:04:43.751622 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:43.721367 23077 consensus_queue.cc:1059] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20251023 02:04:43.737704 23075 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.729053 23006 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.743510 23086 consensus_queue.cc:260] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.753643 23086 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.754268 23082 raft_consensus.cc:3055] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.754667 23082 raft_consensus.cc:740] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:43.754987 22958 raft_consensus.cc:3055] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.755352 22958 raft_consensus.cc:740] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:43.751523 22000 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.756964 22000 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.757850 22077 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.756134 22958 consensus_queue.cc:260] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.756314 23005 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.762897 23055 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:43.765142 23055 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:43.767335 23075 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.768744 23077 raft_consensus.cc:3055] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.771212 23070 raft_consensus.cc:3055] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.771689 23070 raft_consensus.cc:740] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:43.774346 23060 raft_consensus.cc:3055] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.769230 23077 raft_consensus.cc:740] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:43.774822 23060 raft_consensus.cc:740] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:43.733899 23081 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.759620 23023 raft_consensus.cc:2749] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.778527 22077 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.779661 22958 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.781013 23073 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.786271 23075 raft_consensus.cc:2749] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.802942 22077 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.733278 23087 consensus_queue.cc:1059] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20251023 02:04:43.806535 23087 raft_consensus.cc:3055] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.806954 23087 raft_consensus.cc:740] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:43.810438 22077 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.809027 23077 consensus_queue.cc:260] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.785429 23047 consensus_queue.cc:260] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.772696 23070 consensus_queue.cc:260] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.755445 23082 consensus_queue.cc:260] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.828132 22077 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.832307 21857 catalog_manager.cc:5649] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 3 to 4, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:43.833173 22077 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.810846 23075 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.838697 23056 raft_consensus.cc:2749] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.839552 22077 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.837963 22000 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:43.843537 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:43.816385 23055 raft_consensus.cc:2749] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.846436 22000 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.817724 23078 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.847692 22000 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.847692 22077 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.834067 21857 catalog_manager.cc:5649] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 2, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 2 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:43.856715 22958 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.855034 22000 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.813051 23087 consensus_queue.cc:260] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.846383 23023 raft_consensus.cc:2749] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.840394 23033 raft_consensus.cc:2749] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.874159 22618 raft_consensus.cc:1240] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:43.835937 23035 consensus_queue.cc:1059] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251023 02:04:43.852569 23005 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.851953 23018 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.856369 23084 raft_consensus.cc:2749] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.861301 23077 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.863150 23082 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.861629 22000 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.865850 23065 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251023 02:04:43.885886 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 190909us
W20251023 02:04:43.895663 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:43.895287 22000 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.904067 21925 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:43.910100 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 175550us
I20251023 02:04:43.884721 23047 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.888734 23073 raft_consensus.cc:2749] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.855716 23060 consensus_queue.cc:260] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:43.915675 22077 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.913383 23088 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:43.904446 23087 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:43.918834 22000 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.912384 23084 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.922947 22077 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.886443 23056 consensus_queue.cc:1059] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
W20251023 02:04:43.943063 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 216321us
I20251023 02:04:43.913302 23074 consensus_queue.cc:1059] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20251023 02:04:43.946156 22000 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.969597 22077 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:43.970243 21928 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.982676 22000 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.982925 22958 raft_consensus.cc:3055] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:43.983284 22958 raft_consensus.cc:740] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:43.983601 22000 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.984764 22846 raft_consensus.cc:2749] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.985697 21928 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:43.915131 23005 consensus_queue.cc:1059] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
W20251023 02:04:44.041849 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 56345.3us
I20251023 02:04:44.044246 23090 raft_consensus.cc:2749] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.918395 23078 consensus_queue.cc:1059] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.115s
I20251023 02:04:43.920809 23081 consensus_queue.cc:1059] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.113s
I20251023 02:04:43.945804 23075 raft_consensus.cc:2749] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.943173 23076 consensus_queue.cc:1059] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20251023 02:04:43.922283 23018 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:43.917483 23035 consensus_queue.cc:1059] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20251023 02:04:43.945498 23088 raft_consensus.cc:697] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:43.971413 23084 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:44.050120 22958 consensus_queue.cc:260] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:43.984457 23023 raft_consensus.cc:2749] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:44.043131 23069 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251023 02:04:44.052637 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 70105.3us
I20251023 02:04:44.053661 22846 raft_consensus.cc:2749] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251023 02:04:44.053880 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 68262.6us
I20251023 02:04:44.056965 21858 catalog_manager.cc:5649] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 2, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 2 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:04:43.942948 23055 consensus_queue.cc:1059] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.107s
I20251023 02:04:43.981930 23056 consensus_queue.cc:1059] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
W20251023 02:04:44.063375 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 93945.8us
I20251023 02:04:44.046629 23099 tablet_replica.cc:333] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0: stopping tablet replica
W20251023 02:04:44.066637 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:44.072716 21928 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:44.070816 22958 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.069504 22077 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:44.069418 21858 catalog_manager.cc:5649] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 2, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 2 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:04:44.079623 22846 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:44.079852 21928 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:44.080610 22846 consensus_queue.cc:1059] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:44.081523 23067 raft_consensus.cc:3055] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.081904 23067 raft_consensus.cc:740] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:44.082747 23067 consensus_queue.cc:260] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.083863 23067 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.091331 22846 consensus_queue.cc:1059] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:44.090561 23006 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:44.077244 23018 raft_consensus.cc:2749] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:44.102820 23074 raft_consensus.cc:3055] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.076474 23092 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:04:44.122443 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:44.097455 23075 raft_consensus.cc:3055] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
W20251023 02:04:44.130703 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 47169.8us
I20251023 02:04:44.129045 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:44.132699 23090 raft_consensus.cc:3055] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.133066 23090 raft_consensus.cc:740] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:44.133829 22109 raft_consensus.cc:1240] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.103314 23068 consensus_queue.cc:1059] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.084s
I20251023 02:04:44.188822 22104 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.111711 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:44.111511 23074 raft_consensus.cc:740] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
W20251023 02:04:44.192907 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 95673.9us
I20251023 02:04:44.135027 22102 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.133955 23078 raft_consensus.cc:3055] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.134145 23075 raft_consensus.cc:740] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:44.133965 23018 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:44.201108 23032 raft_consensus.cc:3055] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.207876 23032 raft_consensus.cc:740] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:44.208676 23032 consensus_queue.cc:260] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.199113 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:44.229733 23005 raft_consensus.cc:3055] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.230180 23005 raft_consensus.cc:740] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:44.187661 23102 consensus_queue.cc:1059] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.131s
I20251023 02:04:44.232177 21958 raft_consensus.cc:1275] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:44.232513 23102 raft_consensus.cc:3055] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.232339 23005 consensus_queue.cc:260] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.232861 23102 raft_consensus.cc:740] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:44.201167 23055 consensus_queue.cc:1059] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251023 02:04:44.203472 23078 raft_consensus.cc:740] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:44.238694 23070 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.241590 23033 raft_consensus.cc:3055] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.241652 23110 raft_consensus.cc:3055] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.239396 23084 consensus_queue.cc:1048] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20251023 02:04:44.244509 23110 raft_consensus.cc:740] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:44.245627 22110 raft_consensus.cc:1240] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.249694 23056 consensus_queue.cc:1059] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:44.250000 23110 consensus_queue.cc:260] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.250357 23019 consensus_queue.cc:1059] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:44.250993 22958 raft_consensus.cc:3055] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.222290 23090 consensus_queue.cc:260] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.251202 23110 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.251397 23019 raft_consensus.cc:3055] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.251371 22958 raft_consensus.cc:740] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:44.252785 22958 consensus_queue.cc:260] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.251833 23019 raft_consensus.cc:740] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:44.254781 23055 raft_consensus.cc:3055] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.254911 23005 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.255172 23055 raft_consensus.cc:740] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:44.255290 22958 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.255436 23019 consensus_queue.cc:260] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.256620 23019 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.257272 21958 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:44.189070 23088 consensus_queue.cc:237] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.245236 23033 raft_consensus.cc:740] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:44.262929 22110 raft_consensus.cc:1240] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.252279 21954 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251023 02:04:44.286980 23102 consensus_queue.cc:260] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.293469 21958 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20251023 02:04:44.294556 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 250125us
I20251023 02:04:44.243232 23018 consensus_queue.cc:1059] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20251023 02:04:44.265331 23077 consensus_queue.cc:1059] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:44.296650 22000 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:44.307396 22000 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:44.251722 23112 consensus_queue.cc:1059] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.113s
I20251023 02:04:44.309948 23072 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:44.310320 23100 consensus_queue.cc:1059] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
W20251023 02:04:44.311805 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:44.313148 22000 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:44.314465 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:44.316522 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 234641us
I20251023 02:04:44.317078 22076 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:44.322816 22958 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:44.323326 22958 raft_consensus.cc:697] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:44.304556 23060 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.237915 23074 consensus_queue.cc:260] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.302704 23068 consensus_queue.cc:1059] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251023 02:04:44.194092 23092 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.316241 23084 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:44.340040 22958 consensus_queue.cc:237] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.328456 23071 raft_consensus.cc:3055] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.327147 23002 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20251023 02:04:44.308595 23078 consensus_queue.cc:260] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.339582 23003 raft_consensus.cc:3055] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.363456 23006 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:44.365818 23006 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.367681 23006 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:44.360615 23114 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20251023 02:04:44.346920 23071 raft_consensus.cc:740] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:44.370246 23071 consensus_queue.cc:260] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:44.371611 23071 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.378448 21959 raft_consensus.cc:1240] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.253109 23075 consensus_queue.cc:260] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.374001 23119 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
W20251023 02:04:44.395213 22029 consensus_meta.cc:287] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.713s user 0.012s sys 0.000s
I20251023 02:04:44.404047 22029 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
W20251023 02:04:44.405529 23082 consensus_meta.cc:287] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.502s user 0.008s sys 0.001s
I20251023 02:04:44.405640 23033 consensus_queue.cc:260] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.407357 23033 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.409698 23111 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:44.364387 23003 raft_consensus.cc:740] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:04:44.416394 23091 log.cc:927] Time spent T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20251023 02:04:44.396879 23078 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.429597 23032 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.429764 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:44.430608 22615 raft_consensus.cc:2393] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:44.402042 22105 raft_consensus.cc:1275] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:44.402228 22028 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.436338 21958 raft_consensus.cc:1240] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.438853 23106 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:44.389905 23017 consensus_queue.cc:1059] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20251023 02:04:44.427390 23085 consensus_meta.cc:287] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.773s user 0.006s sys 0.005s
I20251023 02:04:44.439523 23106 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.429665 23100 raft_consensus.cc:3055] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.452309 21954 raft_consensus.cc:1240] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.472342 22108 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.478646 22034 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
W20251023 02:04:44.462222 23047 consensus_meta.cc:287] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.521s user 0.007s sys 0.003s
I20251023 02:04:44.481464 23109 consensus_queue.cc:1059] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251023 02:04:44.454244 23100 raft_consensus.cc:740] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:44.483042 22105 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.454113 23003 consensus_queue.cc:260] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:44.488674 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:44.466269 23084 consensus_queue.cc:1059] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251023 02:04:44.497090 22033 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.544739 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:44.499429 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:44.552771 22032 raft_consensus.cc:1240] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.553381 23004 raft_consensus.cc:3055] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.553797 23004 raft_consensus.cc:740] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:44.495538 23100 consensus_queue.cc:260] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.554634 21928 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:44.504597 23092 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:44.543128 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:44.567413 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:44.570950 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:44.571019 22109 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:44.565289 23026 raft_consensus.cc:2749] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:44.571890 22111 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:44.579360 23107 raft_consensus.cc:3055] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.579752 23107 raft_consensus.cc:740] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:44.579917 23100 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.548233 23119 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:44.583891 21928 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:44.581040 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:44.581851 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:44.587291 21925 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:04:44.588820 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:44.588784 22846 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:44.554616 23004 consensus_queue.cc:260] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.591825 22846 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:44.592341 22846 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:04:44.592525 22031 consensus_meta.cc:287] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.541s user 0.008s sys 0.004s
I20251023 02:04:44.592964 22031 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:44.569820 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:44.597376 23023 raft_consensus.cc:3055] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.597782 23023 raft_consensus.cc:740] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:44.597889 21954 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:44.598589 23023 consensus_queue.cc:260] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.602304 23023 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.584504 23084 consensus_queue.cc:1048] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20251023 02:04:44.603211 23119 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.618804 23119 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:44.583670 23092 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:44.619732 23092 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.620421 23106 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:44.622376 23092 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:44.625108 23102 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.626037 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1017) took 1032 ms. Trace:
I20251023 02:04:44.624251 22846 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.630074 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:44.633040 22846 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:44.633854 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:44.626755 22028 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:44.642894 22108 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:44.594888 23068 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:44.644055 22037 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:44.647228 21954 raft_consensus.cc:1240] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.657232 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:44.657308 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:44.658600 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:44.660760 22108 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:44.661638 23108 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:44.656286 22105 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:44.626454 22029 rpcz_store.cc:276] 1023 02:04:43.593213 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:43.597441 (+ 4228us) service_pool.cc:225] Handling call
1023 02:04:44.625857 (+1028416us) spinlock_profiling.cc:243] Waited 77.6 ms on lock 0x7b540020add4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:44.625976 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":178389248}
I20251023 02:04:44.654182 23068 raft_consensus.cc:697] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:44.670614 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:44.674206 23035 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:44.674937 23004 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.675556 23107 consensus_queue.cc:260] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.682245 23107 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.687459 21928 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:44.670001 23092 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:04:44.683696 22028 raft_consensus.cc:2393] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:44.662222 23108 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.697773 23119 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:44.699282 23108 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:44.699970 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:44.703379 23068 consensus_queue.cc:237] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.689944 23089 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:44.714713 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:44.685125 23035 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.693385 23032 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:44.717835 22622 raft_consensus.cc:2393] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:44.716761 23089 raft_consensus.cc:697] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:44.724380 23089 consensus_queue.cc:237] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.733681 23032 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:44.740996 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:44.718540 22103 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
W20251023 02:04:44.745668 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 189915us
I20251023 02:04:44.755189 22077 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:44.756908 21925 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:44.781828 23035 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:44.782167 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:44.782327 23090 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.782640 23111 raft_consensus.cc:3055] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.782900 22033 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:44.782997 23111 raft_consensus.cc:740] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:44.798560 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:44.799010 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:44.776829 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:44.803270 22077 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:44.799175 22033 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:44.759542 23078 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:44.805001 22077 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:44.799021 23085 raft_consensus.cc:3055] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.777781 23068 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:44.778517 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:44.810729 23108 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:44.725096 23055 consensus_queue.cc:260] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.810395 23085 raft_consensus.cc:740] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:44.813815 21954 raft_consensus.cc:2393] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:44.816138 22958 consensus_queue.cc:1059] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:44.800184 23035 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:44.817255 23068 raft_consensus.cc:697] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:44.823724 23035 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.824618 23068 consensus_queue.cc:237] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.800133 23032 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:44.836345 23122 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:44.814270 23033 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:44.837462 22622 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.836952 23122 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.842180 23033 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:44.844609 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:44.834766 23111 consensus_queue.cc:260] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:44.861753 23124 log.cc:927] Time spent T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20251023 02:04:44.862932 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:44.863138 22590 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:44.863756 23087 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:44.859760 23085 consensus_queue.cc:260] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.874926 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:44.864382 23033 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.845327 23099 raft_consensus.cc:2243] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251023 02:04:44.875779 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 268282us
I20251023 02:04:44.877127 23092 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.879040 22077 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:44.873569 23035 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:44.888379 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:44.885996 23032 raft_consensus.cc:2749] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:04:44.904064 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:44.904721 21957 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:44.906628 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:44.906924 23005 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:44.907632 23005 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.909962 23005 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:44.911154 22033 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.911832 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:44.891825 23111 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.883044 23087 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.892376 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:44.925230 22109 raft_consensus.cc:1275] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:44.921774 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:44.932886 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:44.930162 23085 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.934580 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:44.936597 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:44.922818 23119 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:44.940733 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:44.931236 23032 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:44.937593 23047 raft_consensus.cc:3055] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:44.951670 22033 raft_consensus.cc:1275] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:44.954386 23056 consensus_queue.cc:1048] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:44.932570 23122 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:44.947948 23074 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.950778 22589 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:44.955791 22002 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:44.939723 23089 consensus_queue.cc:1048] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
W20251023 02:04:44.966320 23128 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.186s user 0.002s sys 0.002s
I20251023 02:04:44.952582 23047 raft_consensus.cc:740] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:44.952298 23003 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.968202 22622 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 3. (term mismatch)
I20251023 02:04:44.955812 23075 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:44.975330 21928 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:44.978560 22622 pending_rounds.cc:85] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting all ops after (but not including) 2
I20251023 02:04:44.978031 22108 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 3. (term mismatch)
I20251023 02:04:44.982795 22108 pending_rounds.cc:85] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 2
I20251023 02:04:44.983073 22108 pending_rounds.cc:107] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20251023 02:04:44.983382 22108 raft_consensus.cc:2917] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20251023 02:04:44.983742 22108 raft_consensus.cc:2889] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:04:44.966414 21955 raft_consensus.cc:1275] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:44.986248 23068 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:44.993438 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:44.982300 21954 raft_consensus.cc:1240] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:44.955557 23032 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:44.945736 23119 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:44.992816 23087 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:44.993825 23056 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:44.990401 22622 pending_rounds.cc:107] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20251023 02:04:45.002010 23078 consensus_queue.cc:1048] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:44.970767 23122 raft_consensus.cc:2706] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Received pre-election callback for term 2 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20251023 02:04:45.013253 22622 raft_consensus.cc:2917] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20251023 02:04:45.016815 22622 raft_consensus.cc:2889] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:04:45.015522 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:45.020431 23119 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.022272 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.023072 22108 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:45.022593 21928 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:44.973196 23047 consensus_queue.cc:260] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.010980 23068 raft_consensus.cc:697] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:45.027591 23004 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:04:45.040001 23132 log.cc:927] Time spent T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.259s user 0.001s sys 0.000s
I20251023 02:04:45.029778 23111 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
W20251023 02:04:45.035585 23125 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.281s user 0.001s sys 0.002s
I20251023 02:04:45.034029 23101 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:45.040426 23122 consensus_queue.cc:1059] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20251023 02:04:45.047947 23068 consensus_queue.cc:237] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.017342 23056 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.028713 22587 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.059398 23002 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:45.059960 23002 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:45.060361 23032 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.060395 23002 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:04:45.058981 22031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1021) took 1462 ms (client timeout 1599 ms). Trace:
W20251023 02:04:45.064373 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 107733us
I20251023 02:04:45.069240 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.069887 22028 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:45.075102 22077 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.075989 23032 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:45.076488 23032 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:45.076957 23032 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:45.079732 23002 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.081310 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.082607 21957 raft_consensus.cc:2393] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:45.085666 23056 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.086449 23002 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.086663 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.087332 21956 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:45.088416 22076 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.088140 23033 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.063071 23142 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.094583 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.062235 23101 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.106771 23033 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.108430 23078 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:45.108932 23078 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.109434 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:45.069967 23060 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20251023 02:04:45.062075 22031 rpcz_store.cc:269] 1023 02:04:43.594400 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:43.601186 (+ 6786us) service_pool.cc:225] Handling call
1023 02:04:45.055527 (+1454341us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b540040d8d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:45.056648 (+ 1121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":881811840}
I20251023 02:04:45.095400 22033 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:45.110268 21956 raft_consensus.cc:2393] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:45.114001 23078 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.110940 23142 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.114575 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:45.115000 23142 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.116781 21928 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:45.114596 23092 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:45.118938 23099 raft_consensus.cc:2272] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 02:04:45.119371 23055 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:04:45.063868 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 122232us
I20251023 02:04:45.124646 23082 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:45.124696 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:45.120349 23056 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.125520 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.127058 22028 raft_consensus.cc:2393] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:45.144039 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.123425 23085 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.150640 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.151376 22033 raft_consensus.cc:2393] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:45.148330 21957 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:45.149255 22105 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:45.166271 23032 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.073241 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:45.158756 21859 catalog_manager.cc:5649] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 2, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 2 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:45.159404 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.128834 23092 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:45.166844 23081 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.156754 21857 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56864 (request call id 219) took 1345 ms. Trace:
I20251023 02:04:45.155958 23142 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:45.189568 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:45.194936 23075 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:45.147504 23082 raft_consensus.cc:697] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:45.199481 22111 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:45.201491 22031 raft_consensus.cc:2393] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:45.145002 23056 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.141937 23069 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:45.203984 22000 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:45.207221 23056 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.200104 23075 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.211329 23065 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.212917 23005 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.224263 23099 ts_tablet_manager.cc:1916] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251023 02:04:45.200786 23081 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.206921 22029 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:45.229703 23005 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.197078 21857 rpcz_store.cc:276] 1023 02:04:43.811785 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:43.811930 (+ 145us) service_pool.cc:225] Handling call
1023 02:04:45.156651 (+1344721us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:04:43.945968 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:04:44.070886 (+124918us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:04:44.070932 (+ 46us) write_op.cc:435] Acquired schema lock
1023 02:04:44.070960 (+ 28us) tablet.cc:660] Decoding operations
1023 02:04:44.071532 (+ 572us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:04:44.071590 (+ 58us) write_op.cc:641] Partition lock acquired for write op
1023 02:04:44.071615 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
1023 02:04:44.132677 (+ 61062us) tablet.cc:699] Row locks acquired
1023 02:04:44.132698 (+ 21us) write_op.cc:260] PREPARE: finished
1023 02:04:44.132840 (+ 142us) write_op.cc:270] Start()
1023 02:04:44.132952 (+ 112us) write_op.cc:276] Timestamp: P: 1761185084132805 usec, L: 0
1023 02:04:44.132972 (+ 20us) op_driver.cc:348] REPLICATION: starting
1023 02:04:44.210276 (+ 77304us) log.cc:844] Serialized 972 byte log entry
1023 02:04:44.691714 (+481438us) op_driver.cc:464] REPLICATION: finished
1023 02:04:44.834004 (+142290us) write_op.cc:301] APPLY: starting
1023 02:04:44.838462 (+ 4458us) tablet.cc:1365] starting BulkCheckPresence
1023 02:04:44.845716 (+ 7254us) tablet.cc:1368] finished BulkCheckPresence
1023 02:04:44.861915 (+ 16199us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:04:44.864622 (+ 2707us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:04:44.864832 (+ 210us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 02:04:44.864892 (+ 60us) write_op.cc:312] APPLY: finished
1023 02:04:44.865137 (+ 245us) log.cc:844] Serialized 28 byte log entry
1023 02:04:44.865324 (+ 187us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:44.865413 (+ 89us) write_op.cc:454] Released schema lock
1023 02:04:44.865463 (+ 50us) write_op.cc:341] FINISH: Updating metrics
1023 02:04:44.865848 (+ 385us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:44.865873 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4412,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":120837,"apply.run_cpu_time_us":2438,"apply.run_wall_time_us":38380,"num_ops":2,"prepare.queue_time_us":89,"prepare.run_cpu_time_us":4059,"prepare.run_wall_time_us":283403,"replication_time_us":558578,"thread_start_us":46591,"threads_started":1}]]}
I20251023 02:04:45.234350 23075 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:45.234963 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.235608 23032 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.226560 23065 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.236485 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:45.239110 23005 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:45.211362 23069 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.201874 23092 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.148593 22589 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:45.247213 22077 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:45.169081 23085 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.269183 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:45.269815 22618 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:45.270680 23047 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:45.271245 23142 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.273414 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:45.274024 22111 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.277896 22001 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:04:45.280037 23130 log.cc:927] Time spent T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.425s user 0.003s sys 0.001s
I20251023 02:04:45.279659 23056 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.280582 22105 raft_consensus.cc:1275] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:45.249265 23065 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.288595 22033 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:45.289706 22111 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:45.290733 22077 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.248239 23141 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.249436 22031 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:45.265534 23032 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:45.290630 23004 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.299737 22077 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.296985 23005 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.300550 22587 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.301527 23033 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:45.301759 23142 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEARNER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:45.302030 23033 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:45.302438 23146 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:45.302487 23033 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:45.291905 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.304519 23069 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.294500 23075 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:45.306062 23075 raft_consensus.cc:697] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:45.306639 23092 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.307030 23075 consensus_queue.cc:237] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.308271 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.309424 22033 raft_consensus.cc:2393] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:45.314036 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:45.313355 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.314963 22033 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.262866 23082 consensus_queue.cc:237] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:45.295380 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 163844us
I20251023 02:04:45.321380 22111 raft_consensus.cc:2393] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:45.299563 23032 raft_consensus.cc:697] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:45.324239 22031 raft_consensus.cc:1275] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:45.341378 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:45.320992 23103 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.341879 23032 consensus_queue.cc:237] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.342145 23103 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.326067 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:45.327258 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.346731 21957 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:45.327838 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.323210 23055 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:45.336989 23069 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.338143 23081 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.349825 23055 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.344409 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.352387 22105 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:45.352419 22002 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:45.353221 23081 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.353586 23081 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.353981 23081 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.296339 23141 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.355902 23101 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.356832 22000 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.356251 23141 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.357201 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.357223 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.357816 22034 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:45.339213 22075 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:45.358155 22105 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:45.358778 23120 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:45.359223 22587 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.359556 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.360227 23142 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.360638 23142 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.361012 23142 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.362018 23017 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.362432 23017 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.362761 23017 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.359222 22075 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:45.365892 21954 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:45.340201 22033 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:45.316855 23085 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:45.314994 23056 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.344727 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.372345 22108 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:45.345813 23146 raft_consensus.cc:697] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:45.348796 23068 consensus_queue.cc:1048] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:45.342161 22615 raft_consensus.cc:2393] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:45.349902 23069 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.351886 23055 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.309423 23033 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.368849 22077 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.338824 21858 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33304 (request call id 90) took 1305 ms. Trace:
I20251023 02:04:45.369726 22002 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:45.389614 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.390363 22033 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:45.340415 21856 catalog_manager.cc:5649] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 1 to 2, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 2 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:45.360306 22031 raft_consensus.cc:2393] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:45.309823 23004 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.372839 23120 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:45.375933 23081 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:45.384193 23147 log.cc:927] Time spent T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20251023 02:04:45.396068 21928 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:45.395110 23069 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:45.395453 23056 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:45.395385 23074 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:45.395346 23085 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:45.388181 21858 rpcz_store.cc:276] 1023 02:04:44.033406 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:44.043432 (+ 10026us) service_pool.cc:225] Handling call
1023 02:04:45.338765 (+1295333us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:04:44.252219 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:04:44.288772 (+ 36553us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:04:44.288814 (+ 42us) write_op.cc:435] Acquired schema lock
1023 02:04:44.288833 (+ 19us) tablet.cc:660] Decoding operations
1023 02:04:44.289337 (+ 504us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:04:44.289394 (+ 57us) write_op.cc:641] Partition lock acquired for write op
1023 02:04:44.289437 (+ 43us) tablet.cc:683] Acquiring locks for 2 operations
1023 02:04:44.289823 (+ 386us) tablet.cc:699] Row locks acquired
1023 02:04:44.289836 (+ 13us) write_op.cc:260] PREPARE: finished
1023 02:04:44.289976 (+ 140us) write_op.cc:270] Start()
1023 02:04:44.290072 (+ 96us) write_op.cc:276] Timestamp: P: 1761185084289929 usec, L: 0
1023 02:04:44.290085 (+ 13us) op_driver.cc:348] REPLICATION: starting
1023 02:04:44.328201 (+ 38116us) log.cc:844] Serialized 906 byte log entry
1023 02:04:44.765832 (+437631us) op_driver.cc:464] REPLICATION: finished
1023 02:04:44.866223 (+100391us) write_op.cc:301] APPLY: starting
1023 02:04:44.866318 (+ 95us) tablet.cc:1365] starting BulkCheckPresence
1023 02:04:44.866450 (+ 132us) tablet.cc:1368] finished BulkCheckPresence
1023 02:04:44.866587 (+ 137us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:04:44.866943 (+ 356us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:04:44.867130 (+ 187us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 02:04:44.867190 (+ 60us) write_op.cc:312] APPLY: finished
1023 02:04:44.867408 (+ 218us) log.cc:844] Serialized 28 byte log entry
1023 02:04:44.867589 (+ 181us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:44.867687 (+ 98us) write_op.cc:454] Released schema lock
1023 02:04:44.867735 (+ 48us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":291258,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":87868,"apply.run_cpu_time_us":1808,"apply.run_wall_time_us":1808,"num_ops":2,"prepare.queue_time_us":133593,"prepare.run_cpu_time_us":3283,"prepare.run_wall_time_us":90900,"replication_time_us":462537,"thread_start_us":91831,"threads_started":1}]]}
I20251023 02:04:45.415084 23056 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.419552 23056 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.421801 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.422753 22031 raft_consensus.cc:2393] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:45.423095 23074 raft_consensus.cc:697] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:45.422797 23142 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.424125 22618 raft_consensus.cc:2393] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:45.423938 23074 consensus_queue.cc:237] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.409930 22106 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:45.425473 23089 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.404600 23088 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.426673 23088 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.427016 23088 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.428651 23033 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:45.407464 23090 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.429345 23090 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.431203 23090 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.432109 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.432494 22076 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:45.432669 22108 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:45.433686 22000 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.435992 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.436722 22107 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:45.442006 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.442751 22108 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:45.451725 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.452044 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.452564 23033 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:45.452844 22111 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:45.453096 23033 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:45.453524 23033 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:45.453991 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:45.455243 23153 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20251023 02:04:45.456020 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:45.456518 23004 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.444837 23141 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:45.413551 23085 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:45.457846 23085 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:45.457157 23141 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.460070 23141 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.461833 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 40) took 1021 ms. Trace:
I20251023 02:04:45.462332 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:45.441774 23090 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.463033 23090 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.463371 23090 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.463953 23101 consensus_queue.cc:1048] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:45.451907 23120 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.465080 23120 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.465634 21925 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:45.442461 23067 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.466503 23067 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.466941 23120 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.442054 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.468289 23067 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.466493 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.470894 23101 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:45.471300 23101 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:45.471637 23101 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:45.472198 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.472466 23005 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.472856 22110 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:45.474177 22587 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.472900 23005 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.474785 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:45.475518 21956 raft_consensus.cc:2393] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:45.475929 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:45.476727 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.477747 23060 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.478325 21957 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:45.478518 23078 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.478279 23060 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.479014 23078 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.479741 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:45.480262 21954 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.480352 23060 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.480826 23078 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.477438 22029 raft_consensus.cc:2393] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:45.414135 23103 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.482435 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:45.486330 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.487741 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.473393 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:45.488590 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.489784 22036 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:45.489869 22001 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:45.494660 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.487524 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.498313 22034 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:45.498699 22030 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:45.499948 22077 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.502182 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.510102 21928 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:45.426039 23089 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.513866 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.514443 22035 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:45.462180 21958 rpcz_store.cc:276] 1023 02:04:44.440934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:44.441508 (+ 574us) service_pool.cc:225] Handling call
1023 02:04:44.441885 (+ 377us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:44.444637 (+ 2752us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:44.444680 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:44.451704 (+ 7024us) log.cc:844] Serialized 26 byte log entry
1023 02:04:44.671626 (+219922us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:44.671668 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:44.671728 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:45.279508 (+607780us) spinlock_profiling.cc:243] Waited 4.77 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:04:45.461143 (+181635us) spinlock_profiling.cc:243] Waited 9.65 ms on lock 0x7b24000ed0f8. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c83ffe4 00007f9c5c83e0eb 00007f9c5c8cce49 00007f9c5c8ccad5 00007f9c5c8cc030 00007f9c5f65d498 00007f9c5f65dbf8 00007f9c5f635438 00007f9c5f63081a 00007f9c5f62fc66
1023 02:04:45.461646 (+ 503us) raft_consensus.cc:1685] finished
1023 02:04:45.461666 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:45.461775 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2358,"raft.queue_time_us":5375,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":6517,"spinlock_wait_cycles":33180288,"thread_start_us":170293,"threads_started":1,"wal-append.queue_time_us":372660}
I20251023 02:04:45.515419 22077 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.515650 21928 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:45.508162 22076 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:45.520182 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.520891 22110 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:45.521685 22000 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.489482 23141 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:45.480845 23060 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.522945 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.525643 22109 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:45.528141 21962 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:45.529237 22034 raft_consensus.cc:1275] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:45.525050 23078 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:45.519749 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:45.516700 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:45.535231 21953 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.535387 22618 raft_consensus.cc:2393] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:45.516826 23120 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.515409 23023 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:45.528210 23141 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:45.524091 23056 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.518949 23082 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.542877 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.522814 23071 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.576798 23074 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.578189 23003 consensus_queue.cc:1048] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:45.577371 23074 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.582288 23074 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.528163 23081 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.586186 22110 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:45.533305 23017 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.590196 23085 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.538466 23142 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.548267 23101 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.548074 23141 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:04:45.594301 23149 log.cc:927] Time spent T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20251023 02:04:45.540940 23060 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.547870 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.550959 23082 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.557472 23071 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.599483 23089 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.447554 23146 consensus_queue.cc:237] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.549098 23111 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.548722 23056 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.602553 22589 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:45.542577 23120 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.538940 23055 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.604017 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.607455 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:45.648449 22034 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
W20251023 02:04:45.609247 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 87667.1us
W20251023 02:04:45.605283 23152 log.cc:927] Time spent T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20251023 02:04:45.630788 22033 raft_consensus.cc:2393] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:45.651312 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.651616 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.651891 22108 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:45.635761 23065 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.638265 23060 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:45.640659 22109 raft_consensus.cc:1240] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:45.627005 23078 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:45.629094 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:45.649960 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:45.641745 23082 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.653077 22000 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.642696 23071 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.642203 23142 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.653872 22077 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:45.646932 23056 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.645207 23111 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.659219 22111 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.660725 22076 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:45.650437 23120 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.672310 22110 raft_consensus.cc:1275] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:45.654109 23154 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:45.701177 22103 raft_consensus.cc:1275] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251023 02:04:45.690876 23162 log.cc:927] Time spent T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.052s user 0.001s sys 0.001s
I20251023 02:04:45.662196 22105 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.707247 23085 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.672662 23109 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.698916 22028 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:45.695513 23111 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:04:45.690225 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 71258.9us
I20251023 02:04:45.654405 23055 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.661499 23090 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.716797 22589 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:45.662364 23060 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.730031 23060 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.713802 22111 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:45.702203 23078 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:45.712663 23122 raft_consensus.cc:1064] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:45.731731 22000 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.712551 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:45.709743 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:45.724138 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:45.701903 23142 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.714743 23032 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.745805 23032 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.748117 23004 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.748602 23004 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.749174 23032 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.750743 23090 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.751089 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.751865 22104 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:45.754611 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.755553 22034 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:45.740332 22110 raft_consensus.cc:2393] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:45.740944 23086 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:45.741328 23119 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:45.721282 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:45.757026 22077 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.762212 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.763669 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:45.764214 22109 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.764415 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:45.765945 23078 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:45.766497 23078 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:45.767006 23078 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.768211 23119 raft_consensus.cc:697] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:45.713047 23089 consensus_queue.cc:1059] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20251023 02:04:45.766785 23088 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:45.751885 23158 log.cc:927] Time spent T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20251023 02:04:45.735785 23060 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.769171 23119 consensus_queue.cc:237] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.787585 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:45.792335 22109 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:45.794637 22000 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.769760 22618 raft_consensus.cc:2393] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:45.758858 23055 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:45.761165 23101 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.759642 23017 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:45.769897 23085 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.760839 22615 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:45.782193 23086 raft_consensus.cc:697] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:45.715796 23109 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:45.811863 23165 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:45.816311 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:45.824633 22001 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:45.788115 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 68021.6us
I20251023 02:04:45.786816 23099 ts_tablet_manager.cc:1929] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251023 02:04:45.782671 23141 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.805035 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:45.875243 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:45.876579 22109 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:45.810012 23142 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.797384 23033 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:45.893021 21955 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:45.829348 23065 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.829156 23017 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:45.829892 21956 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:45.870568 23047 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:45.881825 23023 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:45.817415 23085 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:45.904948 23161 log.cc:927] Time spent T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20251023 02:04:45.899291 23099 log.cc:1199] T 422ac2af786e439185d0e762e06ef0a3 P 818907d63f1f4d84ba267ebe6ee837e0: Deleting WAL directory at /tmp/dist-test-taskpyLZ7I/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761184990718903-20804-0/minicluster-data/ts-1-root/wals/422ac2af786e439185d0e762e06ef0a3
I20251023 02:04:45.935648 23004 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:46.114179 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 442058us
I20251023 02:04:46.115306 23074 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251023 02:04:46.115402 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 470975us
W20251023 02:04:46.301477 23082 consensus_meta.cc:287] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.629s user 0.008s sys 0.009s
W20251023 02:04:46.115295 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 320796us
I20251023 02:04:46.271525 23103 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:46.272001 23088 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:45.904853 23092 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
W20251023 02:04:46.274602 23078 consensus_meta.cc:287] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.507s user 0.019s sys 0.006s
W20251023 02:04:46.274703 23120 consensus_meta.cc:287] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.572s user 0.008s sys 0.004s
I20251023 02:04:45.905995 22587 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:45.903635 23090 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:46.275383 23111 consensus_meta.cc:287] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.551s user 0.009s sys 0.013s
I20251023 02:04:45.944959 23086 consensus_queue.cc:237] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.277546 23177 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:46.279644 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 344085us
I20251023 02:04:46.280195 23141 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:46.283279 23056 consensus_meta.cc:287] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.581s user 0.008s sys 0.004s
I20251023 02:04:45.896481 22029 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
W20251023 02:04:46.283671 21953 consensus_meta.cc:287] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.748s user 0.020s sys 0.005s
I20251023 02:04:46.284435 23180 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.277479 23032 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.285725 23060 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:45.904868 23169 raft_consensus.cc:3055] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Stepping down as leader of term 2
W20251023 02:04:46.287940 23154 consensus_meta.cc:287] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.584s user 0.018s sys 0.005s
I20251023 02:04:46.288087 23070 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:45.882810 23035 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:46.293120 22002 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:46.289130 23101 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:46.342720 23032 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:46.360433 23101 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.116221 23165 raft_consensus.cc:697] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:46.321635 23078 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.363817 22034 raft_consensus.cc:2393] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:46.303998 23017 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.302115 23023 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.299714 23047 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.312251 23004 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.322362 23179 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.320044 23175 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.320365 23102 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.336159 23033 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.341739 23180 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.325165 23088 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:46.342392 23099 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38996 (request call id 648) took 2660 ms. Trace:
I20251023 02:04:46.314389 23082 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.395246 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.396628 23082 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.346289 23169 raft_consensus.cc:740] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:46.357620 23176 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.397864 23169 consensus_queue.cc:260] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.398561 23176 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.399111 23169 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:04:46.357738 23071 consensus_meta.cc:287] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.668s user 0.010s sys 0.006s
W20251023 02:04:46.358618 22105 consensus_meta.cc:287] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.645s user 0.012s sys 0.001s
I20251023 02:04:46.332401 21953 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:46.358121 23070 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.346632 23154 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:46.359380 23185 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:46.359143 23035 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.333209 23055 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.369742 23017 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.321275 23092 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.319762 23103 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.323702 23111 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.322767 23090 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.323344 23120 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.338856 23065 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:46.365387 23005 consensus_meta.cc:287] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.890s user 0.010s sys 0.007s
I20251023 02:04:46.332154 23056 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.376737 23171 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.377075 23101 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.415194 23070 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.382213 23085 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:46.382855 23179 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.383800 23175 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.331043 23177 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.363512 23032 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.390937 23033 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.393690 23180 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:46.418632 23055 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.394052 23088 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.392858 23102 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.368822 23109 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.402166 23176 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.410394 23178 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:46.411339 23185 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.408671 22105 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:46.334942 23141 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:46.401175 23142 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:46.375713 23004 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.445901 21956 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:46.377125 23182 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:46.418501 21954 consensus_meta.cc:287] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.938s user 0.016s sys 0.003s
I20251023 02:04:46.452247 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.453326 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:46.524442 22111 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.409651 23071 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.487970 22622 raft_consensus.cc:1240] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:46.394236 23078 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.444677 23065 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.493083 23165 consensus_queue.cc:237] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.444468 23005 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.492383 23032 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:46.492653 23111 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.509975 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:46.391734 23099 rpcz_store.cc:276] 1023 02:04:43.681928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:43.710440 (+ 28512us) service_pool.cc:225] Handling call
1023 02:04:43.913944 (+203504us) ts_tablet_manager.cc:1151] Deleting tablet 422ac2af786e439185d0e762e06ef0a3
1023 02:04:43.945688 (+ 31744us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 02:04:45.107035 (+1161347us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b540001ccd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5faa0ef2 00007f9c63483afc 00007f9c64a93daa 00007f9c64ac4a10 00007f9c64aaafdd 00007f9c64aaaf89 00007f9c64aaaf19 00007f9c64aaaee1 00007f9c64aaa10d 0000000000379254
1023 02:04:45.328240 (+221205us) tablet_metadata.cc:641] Metadata flushed
1023 02:04:45.738927 (+410687us) tablet_metadata.cc:641] Metadata flushed
1023 02:04:46.342304 (+603377us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":728059,"spinlock_wait_cycles":9392896,"tablet-delete.queue_time_us":175497,"thread_start_us":142701,"threads_started":1}
I20251023 02:04:46.493249 23171 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.522003 23017 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.520643 23142 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:46.498507 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:46.522727 23047 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.522583 23141 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.508255 22102 raft_consensus.cc:2393] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:46.530761 23186 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:46.531944 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1053) took 1290 ms. Trace:
I20251023 02:04:46.534528 21954 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
W20251023 02:04:46.511989 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:46.537094 22111 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:46.537986 23173 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.495239 23033 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.538622 23035 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.544579 23078 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:46.546633 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:46.546737 23023 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.546798 23071 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.554580 23142 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.534318 22104 raft_consensus.cc:1275] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:46.537477 23178 raft_consensus.cc:697] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:46.533564 23103 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:46.533530 23177 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.559548 21959 raft_consensus.cc:1275] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:46.519385 23185 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.541113 23060 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:46.544909 23089 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.577028 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:46.576962 23003 consensus_queue.cc:1048] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:46.499784 23122 consensus_queue.cc:237] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.544723 23032 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.578243 23088 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.588236 22031 raft_consensus.cc:2368] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:46.600396 22621 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:46.575768 23101 consensus_queue.cc:1048] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251023 02:04:46.580027 23004 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.584630 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 790) took 1101 ms. Trace:
I20251023 02:04:46.581418 23075 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:46.582319 22618 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:46.586324 23078 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.571755 23182 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.597393 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:46.558866 22028 rpcz_store.cc:276] 1023 02:04:45.241080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:45.242622 (+ 1542us) service_pool.cc:225] Handling call
1023 02:04:45.282662 (+ 40040us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:45.714547 (+431885us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:46.531600 (+817053us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b540001dbd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:46.531848 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":876820224}
W20251023 02:04:46.588322 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 690849us
I20251023 02:04:46.607658 23090 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:46.600242 23092 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.577625 23179 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.617861 23082 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.605962 23175 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:46.634313 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:46.634860 23055 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.640103 23176 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:46.644181 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 65532.6us
W20251023 02:04:46.644536 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 351538us
I20251023 02:04:46.648525 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:46.648821 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:46.676338 22033 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:46.619673 23003 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.627558 23005 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:46.660267 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.683142 22111 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:46.683981 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.684996 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.582003 23186 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.587671 23173 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.687405 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:46.661935 22036 raft_consensus.cc:1275] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:46.688578 22111 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.662392 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:46.649562 23178 consensus_queue.cc:237] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.631304 23035 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.668782 22000 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:46.629837 23171 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.631639 23056 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.614953 23089 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.677371 23177 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.679420 22030 raft_consensus.cc:2393] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:46.679631 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.704630 22106 raft_consensus.cc:2410] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20251023 02:04:46.620106 21954 rpcz_store.cc:276] 1023 02:04:45.474457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:45.479507 (+ 5050us) service_pool.cc:225] Handling call
1023 02:04:46.576061 (+1096554us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:46.686326 22110 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:46.684007 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:46.661598 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.689517 22107 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.707454 22108 raft_consensus.cc:2393] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:46.707067 22033 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:46.708189 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.708616 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:46.689976 21925 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:46.672971 23102 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.647094 23075 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.672739 23060 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:46.633335 23120 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.705752 22000 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:46.734371 21962 raft_consensus.cc:1275] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:46.683740 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:46.680287 23179 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:46.708860 22108 raft_consensus.cc:2393] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:46.691548 22077 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:46.709512 22033 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:46.673328 23082 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:46.707041 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:46.710341 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:46.743063 22030 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.743157 22106 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 3
W20251023 02:04:46.735208 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 51783.9us
W20251023 02:04:46.743992 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 52529.2us
I20251023 02:04:46.695500 23065 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.703047 23047 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.742350 22111 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:46.745196 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.730252 22037 raft_consensus.cc:1275] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:46.682416 23176 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.755336 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:46.759789 21961 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:46.744892 22077 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:46.705966 23072 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251023 02:04:46.750598 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 60717.2us
I20251023 02:04:46.742363 23201 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.751844 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:46.751422 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:46.765765 23102 consensus_queue.cc:1048] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:46.765990 21960 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:46.753628 22108 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:46.702280 23035 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:46.686347 23003 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.730974 23141 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.749662 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:46.777843 21958 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:46.749751 23179 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.758538 22000 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:46.761610 23166 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.767426 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.784137 22108 raft_consensus.cc:2368] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:46.743460 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:46.751595 23190 log.cc:927] Time spent T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20251023 02:04:46.771193 22107 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:46.791378 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:46.792415 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.741787 23060 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.755172 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:46.746174 22029 raft_consensus.cc:1275] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:46.750212 23120 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:46.750308 22028 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.756050 23047 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:46.755067 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:46.810830 21959 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:46.775959 22002 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:46.779633 22077 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:46.710453 23177 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.753206 23006 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:46.781366 22000 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:46.755749 23171 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:46.750044 23033 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.785102 22033 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.821261 21925 leader_election.cc:400] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:46.782263 23183 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:46.821846 23183 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:46.822319 23183 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.823642 22033 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:46.824303 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:46.824791 22033 raft_consensus.cc:2393] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:46.790133 23141 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.825428 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:46.825940 22033 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.825968 23141 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.795215 23089 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.795662 21859 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56872 (request call id 210) took 2113 ms. Trace:
I20251023 02:04:46.793488 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.754094 23198 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:46.802325 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:46.778542 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.765859 23201 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:46.830590 22108 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.830132 22107 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:46.832150 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.778684 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:46.832669 22107 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:46.833441 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.834565 22107 raft_consensus.cc:2368] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:46.835203 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.835776 22107 raft_consensus.cc:2368] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:46.836658 22108 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:46.836406 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.746449 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:46.842067 21956 raft_consensus.cc:2410] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20251023 02:04:46.842242 21925 leader_election.cc:403] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:04:46.842959 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.785907 23090 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:46.785703 23032 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.782471 23055 consensus_queue.cc:1048] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20251023 02:04:46.782434 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:46.848891 21960 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:46.738030 23189 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.829257 23141 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.830034 22033 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:46.830479 23198 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:46.837450 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.810894 22030 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:46.856979 23204 consensus_queue.cc:1048] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251023 02:04:46.811746 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:46.856714 22108 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:46.829892 21859 rpcz_store.cc:276] 1023 02:04:44.682044 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:44.682189 (+ 145us) service_pool.cc:225] Handling call
1023 02:04:46.795578 (+2113389us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:04:45.418370 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:04:45.418908 (+ 538us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:04:45.418943 (+ 35us) write_op.cc:435] Acquired schema lock
1023 02:04:45.418961 (+ 18us) tablet.cc:660] Decoding operations
1023 02:04:45.419287 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:04:45.419365 (+ 78us) write_op.cc:641] Partition lock acquired for write op
1023 02:04:45.419396 (+ 31us) tablet.cc:683] Acquiring locks for 1 operations
1023 02:04:45.419564 (+ 168us) tablet.cc:699] Row locks acquired
1023 02:04:45.419579 (+ 15us) write_op.cc:260] PREPARE: finished
1023 02:04:45.419694 (+ 115us) write_op.cc:270] Start()
1023 02:04:45.419789 (+ 95us) write_op.cc:276] Timestamp: P: 1761185085419656 usec, L: 0
1023 02:04:45.419806 (+ 17us) op_driver.cc:348] REPLICATION: starting
1023 02:04:45.420388 (+ 582us) log.cc:844] Serialized 534 byte log entry
1023 02:04:45.638931 (+218543us) op_driver.cc:464] REPLICATION: finished
1023 02:04:46.320627 (+681696us) write_op.cc:301] APPLY: starting
1023 02:04:46.320738 (+ 111us) tablet.cc:1365] starting BulkCheckPresence
1023 02:04:46.320830 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
1023 02:04:46.320955 (+ 125us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:04:46.321222 (+ 267us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:04:46.321383 (+ 161us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 02:04:46.322180 (+ 797us) write_op.cc:312] APPLY: finished
1023 02:04:46.322383 (+ 203us) log.cc:844] Serialized 22 byte log entry
1023 02:04:46.322536 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:46.322938 (+ 402us) write_op.cc:454] Released schema lock
1023 02:04:46.324053 (+ 1115us) write_op.cc:341] FINISH: Updating metrics
1023 02:04:46.340363 (+ 16310us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:46.343386 (+ 3023us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":422591,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":667344,"apply.run_cpu_time_us":1793,"apply.run_wall_time_us":20039,"num_ops":1,"prepare.queue_time_us":72699,"prepare.run_cpu_time_us":2268,"prepare.run_wall_time_us":2267,"replication_time_us":218983,"thread_start_us":450008,"threads_started":2}]]}
I20251023 02:04:46.842916 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:46.812028 22002 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:46.843628 23089 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.844655 22107 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:46.844997 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:04:46.845479 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 64173.7us
I20251023 02:04:46.845664 21925 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:46.806192 23120 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:46.846287 21958 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:46.804754 23166 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:46.844957 22587 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:46.847801 23182 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.817003 22111 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:46.805297 22103 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:46.810034 23078 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.818019 23084 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:46.864166 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:46.986455 22109 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.861279 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:46.840945 23183 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:46.859489 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:46.845849 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:46.846055 23006 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
W20251023 02:04:46.847952 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 64595.6us
I20251023 02:04:46.934198 22106 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:47.022677 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:46.857865 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:46.850865 23065 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:46.854223 23090 raft_consensus.cc:697] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:46.857146 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:47.037422 23208 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.113s user 0.003s sys 0.000s
I20251023 02:04:47.037448 22029 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:47.037927 22615 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:46.935844 23075 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.932732 23119 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:46.876914 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:46.877467 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:46.862320 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:46.913897 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 101951us
I20251023 02:04:47.057858 22002 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:47.057924 23166 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:47.059027 22002 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:46.853852 23195 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.920895 23196 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:46.854099 23186 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:47.028133 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:47.035818 23071 raft_consensus.cc:2804] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:47.051860 23197 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:47.101193 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:47.102032 22107 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:47.103047 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:47.103659 22107 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
W20251023 02:04:47.030262 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 186332us
I20251023 02:04:47.106046 21925 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:46.982862 23167 log.cc:927] Time spent T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 1.209s user 0.000s sys 0.016s
I20251023 02:04:47.034857 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:47.035091 21958 raft_consensus.cc:2393] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:46.850657 23072 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:47.035929 22000 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:46.926192 21928 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:46.929149 23087 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:46.936565 23003 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:47.003558 23120 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.877055 23047 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.875156 23176 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.875757 23082 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.910498 23060 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.025504 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:47.209949 22111 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:46.936766 23035 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:46.859098 23198 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:46.922191 23141 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:47.027562 23084 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:46.861716 23171 raft_consensus.cc:697] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:46.911135 21956 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:47.228566 23171 consensus_queue.cc:237] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.229535 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.230719 21956 raft_consensus.cc:2410] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20251023 02:04:47.231707 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.046975 23006 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.031333 23185 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:47.033174 23092 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:47.028424 23078 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:47.028210 23032 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:47.048592 23183 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:47.038803 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 229171us
I20251023 02:04:47.022931 23102 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:46.877125 23202 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:47.051364 22109 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:47.186024 23087 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.003274 23068 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:47.002722 23165 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:47.002892 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:47.024531 23017 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:47.026033 22033 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:47.266770 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.267506 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.267911 22033 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:47.268422 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.268813 22033 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:47.269343 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.058013 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:47.060173 22002 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:47.028046 22108 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:46.877609 23205 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:47.033640 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:47.035796 23182 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:04:47.122813 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:47.133859 21925 leader_election.cc:400] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:47.281895 21925 leader_election.cc:403] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20251023 02:04:47.142535 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 216457us
I20251023 02:04:47.180140 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.059156 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:47.209391 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:46.911374 23189 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:47.135208 23186 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:47.037508 23065 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.135684 23072 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:47.140669 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.135466 23169 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:47.133168 23005 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:47.052848 23210 consensus_queue.cc:1048] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.171s
I20251023 02:04:47.140158 23196 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:47.188398 23101 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251023 02:04:47.208405 23212 log.cc:927] Time spent T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.103s user 0.000s sys 0.007s
I20251023 02:04:47.232447 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:47.045572 23075 consensus_queue.cc:1048] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:47.319232 22623 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:47.320567 23075 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:04:47.321157 23213 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20251023 02:04:47.321122 23075 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:47.238227 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1064) took 1657 ms. Trace:
I20251023 02:04:47.060506 23119 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:47.222646 23141 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:47.249560 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 215626us
I20251023 02:04:47.328217 22077 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:47.132217 23071 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:47.214726 23176 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:47.219684 23035 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:47.222095 23084 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:47.142023 23197 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:47.241333 23082 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:47.270641 23047 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:46.910887 23177 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.273059 22033 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:47.178124 23195 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:47.279361 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 219305us
I20251023 02:04:47.345999 22002 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:47.271162 23202 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:47.348842 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.324700 23196 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.297431 23186 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.257558 23092 raft_consensus.cc:697] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:47.253235 21956 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:47.277099 21954 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.277410 23166 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:47.365818 22621 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.297309 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.226073 23090 consensus_queue.cc:237] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:47.288372 23142 consensus_meta.cc:287] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.687s user 0.010s sys 0.000s
I20251023 02:04:47.284433 23122 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:47.259955 23102 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.257063 23078 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:47.293628 23070 consensus_queue.cc:1059] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20251023 02:04:47.284476 23006 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:47.352061 23202 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.257958 23185 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.299489 23072 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.294291 23211 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.271939 23068 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:47.276875 23017 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.284328 22619 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:47.298650 22615 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEARNER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251023 02:04:47.329260 22077 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:47.276729 23165 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.312808 21953 raft_consensus.cc:2393] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:04:47.335248 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 299418us
I20251023 02:04:47.379022 23215 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:47.336390 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:47.306377 23005 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:47.286355 23182 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.311223 23101 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:47.280344 23205 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:47.286600 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 152956us
I20251023 02:04:47.330791 23071 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:47.322940 22104 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:47.368062 21961 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
W20251023 02:04:47.370750 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:47.431715 22028 consensus_meta.cc:287] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.588s user 0.004s sys 0.004s
I20251023 02:04:47.343497 23219 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:47.355249 23047 raft_consensus.cc:697] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:47.256913 23032 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.357779 23060 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:47.349783 23220 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:47.340629 23035 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.369279 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:47.456841 22103 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:47.375169 23142 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:47.457690 23035 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:47.458798 23142 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.425406 23017 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.375494 23122 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.460371 22620 raft_consensus.cc:2393] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:47.461200 23122 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.389344 23092 consensus_queue.cc:237] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:47.462023 23032 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:47.390254 23006 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.391256 23075 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:47.393937 23068 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:47.468135 23182 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.327353 22032 rpcz_store.cc:276] 1023 02:04:45.580388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:45.580508 (+ 120us) service_pool.cc:225] Handling call
1023 02:04:45.580827 (+ 319us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:45.581142 (+ 315us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:45.581176 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:45.581721 (+ 545us) log.cc:844] Serialized 26 byte log entry
1023 02:04:45.653632 (+ 71911us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:45.655180 (+ 1548us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:45.659651 (+ 4471us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:46.936274 (+1276623us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:04:47.237828 (+301554us) raft_consensus.cc:1685] finished
1023 02:04:47.237868 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:47.238152 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":370,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":93,"spinlock_wait_cycles":4625152,"thread_start_us":64626,"threads_started":1,"wal-append.queue_time_us":116875}
I20251023 02:04:47.469108 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.470227 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:47.433329 22028 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:47.470765 22032 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.394989 23195 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.395224 23197 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:47.395797 23198 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:47.398218 23065 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.398432 23141 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.399423 23119 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:47.479871 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.480262 21856 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56856 (request call id 226) took 2746 ms. Trace:
I20251023 02:04:47.401294 21954 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:47.401804 23102 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:46.790107 21847 catalog_manager.cc:4985] TS 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): tablet 422ac2af786e439185d0e762e06ef0a3 (table test-workload [id=6e96b4ccdb4143009781f4a2f459bbb4]) successfully deleted
I20251023 02:04:47.403445 23078 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:47.404810 23089 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:47.408406 23189 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:47.408739 23211 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:47.340773 23176 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:47.348179 23082 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:47.413264 22000 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:47.306736 23169 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:47.378317 23173 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.372342 23214 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:47.415935 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 86766.8us
I20251023 02:04:47.494092 22622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 216) took 1952 ms. Trace:
I20251023 02:04:47.374665 22587 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:47.424405 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:47.425932 21925 leader_election.cc:400] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:47.496483 21925 leader_election.cc:403] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20251023 02:04:47.497067 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 71296.4us
I20251023 02:04:47.431661 21961 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20251023 02:04:47.497753 21961 pending_rounds.cc:85] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 4
I20251023 02:04:47.498040 21961 pending_rounds.cc:107] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20251023 02:04:47.498270 21961 raft_consensus.cc:2889] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251023 02:04:47.372097 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 86532.4us
I20251023 02:04:47.434670 23215 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.434252 23047 consensus_queue.cc:237] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.423791 23216 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:47.434526 23101 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.451359 23185 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.452893 22001 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:47.550670 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 97905.2us
I20251023 02:04:47.453117 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:47.454175 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:47.495000 23179 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:47.456000 23220 raft_consensus.cc:697] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:47.465375 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1076) took 1013 ms. Trace:
I20251023 02:04:47.412408 23087 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.467190 23006 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.471916 22621 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:47.419793 23005 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.434127 23186 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:47.483520 21847 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 772640us
I20251023 02:04:47.484869 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.471469 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:47.487191 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 74027us
I20251023 02:04:47.488500 22037 raft_consensus.cc:2368] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3.
I20251023 02:04:47.489980 23169 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.491721 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55770 (request call id 37) took 1731 ms. Trace:
I20251023 02:04:47.470523 23072 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.514539 23214 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.494365 22622 rpcz_store.cc:276] 1023 02:04:45.536776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:45.536933 (+ 157us) service_pool.cc:225] Handling call
1023 02:04:45.537191 (+ 258us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:46.527890 (+990699us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:47.489241 (+961351us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b5400194e54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:47.489409 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2186396928}
I20251023 02:04:47.517854 21925 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:47.518548 21961 raft_consensus.cc:1275] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:47.521976 23165 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:47.524189 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:47.532274 21959 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:47.489405 23090 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:47.551653 22001 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:47.435964 23060 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:47.555197 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.550727 23092 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:47.480614 21856 rpcz_store.cc:276] 1023 02:04:44.733925 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:44.734829 (+ 904us) service_pool.cc:225] Handling call
1023 02:04:44.960736 (+225907us) spinlock_profiling.cc:243] Waited 4.74 ms on lock 0x7b540042f254. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa02fe0 00007f9c5faa1aca 00007f9c65813bbc 00007f9c6590de7e 00007f9c63e0d4a4 00007f9c63e0d432 00007f9c63e0d361 00007f9c63e0d2dc 00007f9c63e0c592 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:47.480206 (+2519470us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:04:45.420803 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:04:45.421285 (+ 482us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:04:45.421312 (+ 27us) write_op.cc:435] Acquired schema lock
1023 02:04:45.421327 (+ 15us) tablet.cc:660] Decoding operations
1023 02:04:45.440805 (+ 19478us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:04:45.445448 (+ 4643us) write_op.cc:641] Partition lock acquired for write op
1023 02:04:45.446758 (+ 1310us) tablet.cc:683] Acquiring locks for 1 operations
1023 02:04:45.457197 (+ 10439us) tablet.cc:699] Row locks acquired
1023 02:04:45.457212 (+ 15us) write_op.cc:260] PREPARE: finished
1023 02:04:45.457346 (+ 134us) write_op.cc:270] Start()
1023 02:04:45.457446 (+ 100us) write_op.cc:276] Timestamp: P: 1761185085457303 usec, L: 0
1023 02:04:45.457465 (+ 19us) op_driver.cc:348] REPLICATION: starting
1023 02:04:45.541915 (+ 84450us) log.cc:844] Serialized 534 byte log entry
1023 02:04:46.271533 (+729618us) op_driver.cc:464] REPLICATION: finished
1023 02:04:46.365639 (+ 94106us) write_op.cc:301] APPLY: starting
1023 02:04:46.368345 (+ 2706us) tablet.cc:1365] starting BulkCheckPresence
1023 02:04:46.375344 (+ 6999us) tablet.cc:1368] finished BulkCheckPresence
1023 02:04:46.391807 (+ 16463us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:04:46.416921 (+ 25114us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:04:46.445285 (+ 28364us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 02:04:46.480342 (+ 35057us) write_op.cc:312] APPLY: finished
1023 02:04:46.489055 (+ 8713us) log.cc:844] Serialized 22 byte log entry
1023 02:04:46.498308 (+ 9253us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:46.504074 (+ 5766us) write_op.cc:454] Released schema lock
1023 02:04:46.506270 (+ 2196us) write_op.cc:341] FINISH: Updating metrics
1023 02:04:46.542874 (+ 36604us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:46.544815 (+ 1941us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":472217,"reported_tablets":1,"spinlock_wait_cycles":10894080,"child_traces":[["op",{"apply.queue_time_us":90483,"apply.run_cpu_time_us":2562,"apply.run_wall_time_us":169362,"num_ops":1,"prepare.queue_time_us":11798,"prepare.run_cpu_time_us":2623,"prepare.run_wall_time_us":137644,"replication_time_us":813949,"thread_start_us":25773,"threads_started":1}]]}
I20251023 02:04:47.559585 22110 rpcz_store.cc:276] 1023 02:04:46.451910 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:46.452010 (+ 100us) service_pool.cc:225] Handling call
1023 02:04:47.465230 (+1013220us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b5400021554. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:47.465319 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":372640512}
I20251023 02:04:47.526521 23173 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:47.547359 23216 raft_consensus.cc:697] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:47.513705 23120 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:47.577402 22076 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.820s (SENT) [suppressed 5 similar messages]
I20251023 02:04:47.870398 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 808) took 1088 ms. Trace:
I20251023 02:04:47.872028 21961 rpcz_store.cc:276] 1023 02:04:46.781825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:47.499130 (+717305us) service_pool.cc:225] Handling call
1023 02:04:47.499507 (+ 377us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:47.870146 (+370639us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:47.870334 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:47.735577 22032 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:47.750680 21859 catalog_manager.cc:5649] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:47.750327 21858 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56864 (request call id 220) took 2211 ms. Trace:
I20251023 02:04:47.667944 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:47.736235 22035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 807) took 1088 ms. Trace:
W20251023 02:04:47.548043 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 179195us
I20251023 02:04:47.874634 22034 rpcz_store.cc:276] 1023 02:04:45.760668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:45.761876 (+ 1208us) service_pool.cc:225] Handling call
1023 02:04:47.491506 (+1729630us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b5400280854. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:47.491657 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1381355392}
I20251023 02:04:47.978024 21858 rpcz_store.cc:276] 1023 02:04:45.539130 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:45.539457 (+ 327us) service_pool.cc:225] Handling call
1023 02:04:45.580280 (+ 40823us) spinlock_profiling.cc:243] Waited 6.95 ms on lock 0x7b540042f254. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa02fe0 00007f9c5faa1aca 00007f9c65813bbc 00007f9c6590de7e 00007f9c63e0d4a4 00007f9c63e0d432 00007f9c63e0d361 00007f9c63e0d2dc 00007f9c63e0c592 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:47.750249 (+2169969us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":15995136}
I20251023 02:04:47.563796 22624 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:47.555262 23221 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251023 02:04:47.920213 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 402445us
I20251023 02:04:47.876704 22036 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:47.931499 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.939390 22109 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251023 02:04:47.945055 22033 raft_consensus.cc:2368] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:47.878357 22000 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:47.947786 22028 raft_consensus.cc:2393] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:47.978776 22035 rpcz_store.cc:276] 1023 02:04:46.647960 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:46.648488 (+ 528us) service_pool.cc:225] Handling call
1023 02:04:47.735301 (+1086813us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b58002d5a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:04:47.735588 (+ 287us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:47.735820 (+ 232us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:47.735858 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:47.735910 (+ 52us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:47.735952 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:47.736025 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:47.736166 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2619776}
I20251023 02:04:47.987290 21925 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:47.988170 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:47.985721 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 107472us
I20251023 02:04:47.988801 22035 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.989166 22000 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:47.881743 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 330231us
I20251023 02:04:47.989391 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:47.983798 21859 catalog_manager.cc:5649] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 2, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 2 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
W20251023 02:04:47.991901 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:47.992493 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:47.991253 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.870054 23060 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.994354 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55770 (request call id 38) took 1304 ms. Trace:
I20251023 02:04:47.992743 21859 catalog_manager.cc:5649] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 2, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 2 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:47.996337 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.992625 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.997881 21962 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:47.997298 21859 catalog_manager.cc:5649] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 2, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 2 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:47.649976 23198 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:47.672029 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1047) took 1128 ms. Trace:
I20251023 02:04:47.658146 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.999747 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.679884 23201 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:47.671399 22031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 796) took 2242 ms (client timeout 1976 ms). Trace:
W20251023 02:04:47.676069 22102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1070) took 2068 ms (client timeout 1582 ms). Trace:
I20251023 02:04:47.880378 23090 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:47.666445 23166 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:47.974465 23214 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:47.949676 23120 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:48.000428 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.002025 22075 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:47.747215 23171 raft_consensus.cc:3055] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:47.619208 23176 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:47.933729 23070 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:48.006724 22000 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:48.068898 21961 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:47.743573 23005 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.680495 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.999347 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.619767 23084 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.009169 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:47.986572 23221 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.011471 22590 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:48.001338 23227 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20251023 02:04:47.746856 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.012970 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.013510 22002 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:47.626250 23205 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.097302 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
W20251023 02:04:48.097244 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 83816.8us
I20251023 02:04:48.097905 22621 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.990159 23023 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:47.932015 23092 raft_consensus.cc:697] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:47.999226 23165 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:47.994948 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.738823 23101 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.650043 23215 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:47.996733 23170 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election won for term 2
W20251023 02:04:48.069528 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 67591.5us
I20251023 02:04:48.070782 21957 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:48.185540 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1048) took 1437 ms. Trace:
I20251023 02:04:48.185913 21957 rpcz_store.cc:276] 1023 02:04:46.748529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:46.748659 (+ 130us) service_pool.cc:225] Handling call
1023 02:04:47.667743 (+919084us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b5800002a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:04:48.185481 (+517738us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13844352}
I20251023 02:04:48.187086 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.187602 21957 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:48.065026 22033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 818) took 1032 ms. Trace:
I20251023 02:04:47.997339 23056 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:04:48.072858 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 66222.6us
I20251023 02:04:48.002739 21928 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:48.000532 23078 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:48.073832 23072 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.177023 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:47.655267 23071 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.078243 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:48.203143 21961 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.012508 23081 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:48.080114 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.737843 23202 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.205103 21962 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:48.207683 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 812) took 1413 ms. Trace:
I20251023 02:04:48.208263 21962 raft_consensus.cc:1275] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251023 02:04:48.081954 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 70543.5us
I20251023 02:04:48.006052 21925 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:48.210384 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 204427us
I20251023 02:04:47.994810 22028 rpcz_store.cc:276] 1023 02:04:46.690390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:47.471221 (+780831us) service_pool.cc:225] Handling call
1023 02:04:47.994293 (+523072us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:48.190413 23205 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:48.208053 21957 rpcz_store.cc:276] 1023 02:04:46.793959 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.190888 (+1396929us) service_pool.cc:225] Handling call
1023 02:04:48.207096 (+ 16208us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:48.207299 (+ 203us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:48.207348 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:48.207401 (+ 53us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:48.207443 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:48.207511 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:48.207623 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:48.090883 22035 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:48.231066 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:48.220005 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 184260us
I20251023 02:04:48.086020 23221 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.080742 23201 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.096230 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:47.633492 23082 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.072093 23087 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:48.102058 22002 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:04:48.236065 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 134114us
I20251023 02:04:48.072290 23171 raft_consensus.cc:740] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:48.006523 22105 rpcz_store.cc:276] 1023 02:04:46.544032 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:46.546072 (+ 2040us) service_pool.cc:225] Handling call
1023 02:04:46.579654 (+ 33582us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:46.680988 (+101334us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:46.684377 (+ 3389us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:46.688291 (+ 3914us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:46.695867 (+ 7576us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:47.671707 (+975840us) spinlock_profiling.cc:243] Waited 90.4 ms on lock 0x7b540032fad4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:47.671782 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:47.671940 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":207842048}
I20251023 02:04:48.077145 23214 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.065737 21954 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:47.654706 23196 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:47.973359 23169 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.096711 23084 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.069686 23090 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:04:48.015214 22031 rpcz_store.cc:269] 1023 02:04:45.428841 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:45.440733 (+ 11892us) service_pool.cc:225] Handling call
1023 02:04:47.671112 (+2230379us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540020b2d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:47.671287 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2397313920}
I20251023 02:04:47.950145 23216 consensus_queue.cc:237] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:48.131587 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 554288us
W20251023 02:04:48.115279 23229 log.cc:927] Time spent T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.443s user 0.004s sys 0.009s
I20251023 02:04:48.236780 23227 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
W20251023 02:04:48.064888 22102 rpcz_store.cc:269] 1023 02:04:45.607204 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:45.608673 (+ 1469us) service_pool.cc:225] Handling call
1023 02:04:46.395090 (+786417us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:04:47.675805 (+1280715us) spinlock_profiling.cc:243] Waited 90.5 ms on lock 0x7b54001dbd54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:47.675975 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":214699008}
I20251023 02:04:48.242141 23214 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.089134 21955 raft_consensus.cc:2376] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3. Re-sending same reply.
I20251023 02:04:48.104461 23047 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.102720 23168 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:48.170007 23198 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:48.168759 22037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55770 (request call id 40) took 1472 ms (client timeout 1675 ms). Trace:
I20251023 02:04:48.137369 21960 raft_consensus.cc:2393] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:48.104399 22029 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:48.188177 22033 rpcz_store.cc:276] 1023 02:04:47.032918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:47.348540 (+315622us) service_pool.cc:225] Handling call
1023 02:04:48.064831 (+716291us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b540026f554. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:48.064983 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1369340160}
I20251023 02:04:48.189461 22000 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:48.233198 23092 consensus_queue.cc:237] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.268445 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:48.193663 21959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 809) took 1411 ms (client timeout 1524 ms). Trace:
I20251023 02:04:48.074008 23070 raft_consensus.cc:697] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:48.256575 23214 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:47.560043 23220 consensus_queue.cc:237] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.211395 23081 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:48.277762 23231 log.cc:927] Time spent T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.398s user 0.003s sys 0.000s
I20251023 02:04:48.012768 23178 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.064308 23166 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:48.190122 23215 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:48.223248 22106 raft_consensus.cc:2368] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3.
I20251023 02:04:48.221300 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1087) took 1449 ms. Trace:
I20251023 02:04:48.290567 22107 rpcz_store.cc:276] 1023 02:04:46.772023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:47.105460 (+333437us) service_pool.cc:225] Handling call
1023 02:04:47.118282 (+ 12822us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:47.246297 (+128015us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:47.246344 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:47.246976 (+ 632us) log.cc:844] Serialized 26 byte log entry
1023 02:04:47.740646 (+493670us) spinlock_profiling.cc:243] Waited 4.36 ms on lock 0x7b18002eb700. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9cdd37 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c5f62f2bb 00007f9c5f634dc3 00007f9c5f635171 00007f9c5fa5465f
1023 02:04:47.993681 (+253035us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:47.993718 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:47.993776 (+ 58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:48.212263 (+218487us) raft_consensus.cc:1685] finished
1023 02:04:48.212284 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:48.221218 (+ 8934us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13051,"raft.queue_time_us":141566,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":15028,"spinlock_wait_cycles":10017152,"thread_start_us":367242,"threads_started":2,"wal-append.queue_time_us":868456}
I20251023 02:04:48.205019 23170 raft_consensus.cc:697] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:48.293566 23170 consensus_queue.cc:237] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.213675 21925 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:48.296733 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 83191.7us
I20251023 02:04:48.297780 23026 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.218218 23175 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.232036 22035 raft_consensus.cc:2393] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.298152 23026 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.298384 23175 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.298537 23026 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.298760 23175 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.299077 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.300315 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.301126 22035 raft_consensus.cc:2368] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:48.092360 22036 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:48.302350 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.303228 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 812) took 1654 ms. Trace:
I20251023 02:04:48.298152 21925 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:48.303928 23211 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:48.304369 23120 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:48.195031 23078 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.304458 23211 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.305081 23082 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.304929 23120 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.305485 23082 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.305922 23082 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.306433 23078 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:48.306562 23211 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.238786 21953 raft_consensus.cc:1275] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251023 02:04:48.239192 23232 log.cc:927] Time spent T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.134s user 0.002s sys 0.000s
I20251023 02:04:48.303650 22036 rpcz_store.cc:276] 1023 02:04:46.649241 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.302079 (+1652838us) service_pool.cc:225] Handling call
1023 02:04:48.303167 (+ 1088us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:48.223243 23005 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.239351 23087 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.201136 23056 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.327358 23056 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:48.328490 23070 consensus_queue.cc:237] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.223593 23205 raft_consensus.cc:697] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:48.329407 23166 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.246865 21957 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251023 02:04:48.330235 23205 consensus_queue.cc:237] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.235657 23201 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.331348 23201 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.331879 21957 raft_consensus.cc:1275] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:48.332304 23202 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.243543 22002 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:48.334510 22033 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:48.334697 23087 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:48.334944 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 91454.7us
I20251023 02:04:48.168094 22621 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:48.335842 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.336663 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:48.337342 22622 raft_consensus.cc:2393] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.151582 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1093) took 1342 ms. Trace:
I20251023 02:04:48.254073 22030 raft_consensus.cc:2393] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.244794 23101 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.337692 22109 rpcz_store.cc:276] 1023 02:04:46.809024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:47.256954 (+447930us) service_pool.cc:225] Handling call
1023 02:04:47.257330 (+ 376us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:47.984930 (+727600us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:48.151232 (+166302us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b54001ac7d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:48.151501 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1568292096}
I20251023 02:04:48.347465 22001 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:48.255339 21958 raft_consensus.cc:2393] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:48.257395 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.349141 22110 raft_consensus.cc:2393] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.247131 23072 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:48.250906 23192 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:48.252394 23102 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:04:48.268605 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 79240.9us
I20251023 02:04:48.356628 22000 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:48.346861 23084 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:04:48.189560 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 188371us
I20251023 02:04:48.358827 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:48.311861 23078 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:48.197057 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.256450 23176 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:48.259037 23047 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.361656 23084 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:48.363417 23047 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.363682 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.364415 22034 raft_consensus.cc:2393] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.365029 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.353515 23072 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:48.365664 21928 leader_election.cc:400] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:48.365830 23072 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:48.366544 23026 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.367291 23166 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.367854 23166 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:48.368263 23166 raft_consensus.cc:697] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:48.368428 23026 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.366544 21928 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:48.264564 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.369884 22031 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:48.370093 23169 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.370766 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.371385 22031 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:48.370011 23026 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.371841 23026 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.372184 23026 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.372208 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.374255 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1069) took 1682 ms. Trace:
I20251023 02:04:48.374459 23102 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.374616 22030 rpcz_store.cc:276] 1023 02:04:46.691984 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:46.863895 (+171911us) service_pool.cc:225] Handling call
1023 02:04:48.374079 (+1510184us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54003a8c54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:48.374202 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2826754816}
I20251023 02:04:48.375622 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:48.265139 22037 rpcz_store.cc:269] 1023 02:04:46.690917 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:46.790997 (+100080us) service_pool.cc:225] Handling call
1023 02:04:48.147218 (+1356221us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b540040d3d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:48.162838 (+ 15620us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1593949056}
I20251023 02:04:48.376401 23102 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.376291 23201 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:48.377301 22037 raft_consensus.cc:1275] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:48.377465 22032 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.378067 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1067) took 1686 ms. Trace:
I20251023 02:04:48.378363 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:48.378568 22037 rpcz_store.cc:276] 1023 02:04:46.691436 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.376642 (+1685206us) service_pool.cc:225] Handling call
1023 02:04:48.376970 (+ 328us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:48.377736 (+ 766us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:48.378001 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:48.261034 21955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 800) took 1701 ms (client timeout 1820 ms). Trace:
I20251023 02:04:48.265280 23237 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:48.281445 21961 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:48.382603 23237 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:48.382273 23078 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.383019 23237 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:48.383347 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:48.383927 21961 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.384891 23078 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:48.255409 23227 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.261401 23023 raft_consensus.cc:697] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:48.386039 23023 consensus_queue.cc:237] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.386801 23227 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:48.387244 21958 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 811) took 1593 ms (client timeout 2007 ms). Trace:
W20251023 02:04:48.387840 21958 rpcz_store.cc:269] 1023 02:04:46.793657 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:47.179781 (+386124us) service_pool.cc:225] Handling call
1023 02:04:48.387010 (+1207229us) spinlock_profiling.cc:243] Waited 972 ms on lock 0x7b5400107554. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:48.387165 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2235852032}
I20251023 02:04:48.365751 23176 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.306831 23120 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:48.308641 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.314132 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.256752 23171 consensus_queue.cc:260] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.389699 23081 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.389869 22035 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:48.390025 23171 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.390106 23176 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:48.390401 22035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55770 (request call id 41) took 1561 ms. Trace:
I20251023 02:04:48.390736 22035 rpcz_store.cc:276] 1023 02:04:46.829408 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.313896 (+1484488us) service_pool.cc:225] Handling call
1023 02:04:48.390336 (+ 76440us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:48.264031 23168 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.315127 23196 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.280179 23240 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20251023 02:04:48.315745 22105 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20251023 02:04:48.316783 21960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 810) took 1527 ms (client timeout 1992 ms). Trace:
I20251023 02:04:48.317798 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.318976 23071 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.268914 23198 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.322343 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56894 (request call id 59) took 1016 ms. Trace:
I20251023 02:04:48.410092 22106 rpcz_store.cc:276] 1023 02:04:47.306355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:47.306491 (+ 136us) service_pool.cc:225] Handling call
1023 02:04:48.322146 (+1015655us) spinlock_profiling.cc:243] Waited 807 ms on lock 0x7b54000aa5d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:48.322285 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1857107840}
I20251023 02:04:48.323464 23165 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.324537 23175 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.328611 23060 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.333155 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 49) took 1563 ms. Trace:
I20251023 02:04:48.241112 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:48.334174 22029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 817) took 1301 ms (client timeout 1516 ms). Trace:
I20251023 02:04:48.358071 23192 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.281867 23178 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.281379 23239 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:48.281436 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:48.269026 21959 rpcz_store.cc:269] 1023 02:04:46.782093 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:46.811519 (+ 29426us) service_pool.cc:225] Handling call
1023 02:04:48.193427 (+1381908us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b54000c5654. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:48.193568 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1447378176}
I20251023 02:04:48.419466 23165 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:04:48.380075 21955 rpcz_store.cc:269] 1023 02:04:46.558089 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:46.572247 (+ 14158us) service_pool.cc:225] Handling call
1023 02:04:47.680276 (+1108029us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b5800002a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:04:48.259399 (+579123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5889408}
I20251023 02:04:48.387308 23227 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:48.310628 23215 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.439402 22104 raft_consensus.cc:2393] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.446985 23215 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:48.369071 23166 consensus_queue.cc:237] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.447885 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1068) took 1756 ms. Trace:
I20251023 02:04:48.448283 22032 rpcz_store.cc:276] 1023 02:04:46.691714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.378721 (+1687007us) service_pool.cc:225] Handling call
1023 02:04:48.378997 (+ 276us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:48.446990 (+ 67993us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:48.447028 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:48.447075 (+ 47us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:48.447110 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:48.447666 (+ 556us) spinlock_profiling.cc:243] Waited 67.8 ms on lock 0x7b540034ab54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:48.447711 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:48.447831 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":155951872}
I20251023 02:04:48.373387 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.437076 23178 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:04:48.402678 21960 rpcz_store.cc:269] 1023 02:04:46.789351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:47.057745 (+268394us) service_pool.cc:225] Handling call
1023 02:04:48.316567 (+1258822us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b54000c4254. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:48.316725 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1979761024}
I20251023 02:04:48.363566 23084 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.450526 23090 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.451814 23084 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.341562 21962 raft_consensus.cc:1275] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:48.452855 23090 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.429461 22033 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
W20251023 02:04:48.428206 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 71699.7us
I20251023 02:04:48.410579 23119 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:48.432183 21957 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:48.432367 21961 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
W20251023 02:04:48.427718 22029 rpcz_store.cc:269] 1023 02:04:47.032738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:47.049976 (+ 17238us) service_pool.cc:225] Handling call
1023 02:04:48.334004 (+1284028us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b540027b354. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:48.334120 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1835553024}
I20251023 02:04:48.434088 22028 raft_consensus.cc:2368] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
I20251023 02:04:48.480500 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.481110 22111 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:48.436596 23192 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.347925 23005 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:48.439401 23196 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:48.442092 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:48.443324 23072 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.444005 23026 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.444766 23198 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.445293 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:48.430140 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 125444us
I20251023 02:04:48.462811 22077 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:48.493058 22077 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:48.472841 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.494815 22110 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
W20251023 02:04:48.477327 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:48.479131 22107 raft_consensus.cc:1275] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:48.482545 22033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1090) took 1031 ms. Trace:
I20251023 02:04:48.496851 22033 rpcz_store.cc:276] 1023 02:04:47.451610 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.481145 (+1029535us) service_pool.cc:225] Handling call
1023 02:04:48.481470 (+ 325us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:48.482153 (+ 683us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:48.482214 (+ 61us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:48.482264 (+ 50us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:48.482304 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:48.482369 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:48.482488 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:48.485203 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.499955 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.501447 22111 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:48.502684 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.503446 22104 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:48.489432 23047 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.498831 22033 raft_consensus.cc:1275] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:48.506207 22106 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251023 02:04:48.507494 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.520666 22076 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:48.494184 22077 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:48.522352 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.438810 22002 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:04:48.524053 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:48.528470 22076 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:48.531643 23201 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.531968 23072 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.536015 23237 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.537082 22618 raft_consensus.cc:1240] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:48.490772 23005 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.538435 22029 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251023 02:04:48.539230 23005 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.541953 23214 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.542405 23242 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:48.543663 23214 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.484181 23119 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.544863 22036 raft_consensus.cc:1275] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251023 02:04:48.545259 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 51180us
I20251023 02:04:48.490221 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1073) took 1055 ms. Trace:
I20251023 02:04:48.548134 22077 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:48.555482 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1080) took 1118 ms. Trace:
I20251023 02:04:48.555986 22036 rpcz_store.cc:276] 1023 02:04:47.437175 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.393829 (+956654us) service_pool.cc:225] Handling call
1023 02:04:48.394077 (+ 248us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:48.545199 (+151122us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:48.555186 (+ 9987us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b540040d8d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:48.555412 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":346189184}
I20251023 02:04:48.557530 22034 raft_consensus.cc:2393] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.553941 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.559425 22107 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:48.509706 23176 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:48.562451 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1085) took 1112 ms. Trace:
I20251023 02:04:48.563027 22034 rpcz_store.cc:276] 1023 02:04:47.450022 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.364826 (+914804us) service_pool.cc:225] Handling call
1023 02:04:48.562283 (+197457us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b540026f054. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:48.562398 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":441028352}
I20251023 02:04:48.524590 23215 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.562529 22077 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:48.569051 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.576051 22108 raft_consensus.cc:2393] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:48.570256 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.508877 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1083) took 1070 ms. Trace:
I20251023 02:04:48.576742 23221 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.576982 22032 rpcz_store.cc:276] 1023 02:04:47.438079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.449430 (+1011351us) service_pool.cc:225] Handling call
1023 02:04:48.485542 (+ 36112us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:48.500737 (+ 15195us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:48.508536 (+ 7799us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:48.508602 (+ 66us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:48.508643 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:48.508713 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:48.508826 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:48.572849 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.578372 22111 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:48.521399 22037 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:48.578738 23221 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.579262 23221 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.579706 23221 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.580031 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 42) took 1532 ms. Trace:
I20251023 02:04:48.580410 22037 rpcz_store.cc:276] 1023 02:04:47.047596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.379490 (+1331894us) service_pool.cc:225] Handling call
1023 02:04:48.483821 (+104331us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:48.579027 (+ 95206us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:48.579846 (+ 819us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54003aa550. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:48.579978 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":234658048}
I20251023 02:04:48.582067 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.582813 22105 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.580041 23221 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:04:48.579100 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 140421us
I20251023 02:04:48.590782 23252 consensus_queue.cc:1048] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:48.592028 22105 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:48.594065 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:04:48.597379 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 250033us
I20251023 02:04:48.598786 21928 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:48.599723 23205 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.600113 23205 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.600533 23205 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.599736 21928 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:48.601639 23238 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.601994 23238 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.602330 23238 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.603485 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.605239 22105 raft_consensus.cc:2393] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:48.608898 23238 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.610183 23166 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:48.610405 23023 consensus_queue.cc:1048] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:48.612664 22002 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:48.526127 23220 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:48.614521 22077 leader_election.cc:400] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:48.615537 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.621888 22077 leader_election.cc:403] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:04:48.549469 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.556500 23120 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:48.556084 23078 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.491818 23196 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.531983 23060 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.561270 23214 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:48.626932 22002 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:48.572729 23186 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.562100 23175 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.562892 23176 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:48.559962 22033 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38996
I20251023 02:04:48.574227 23215 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.631139 22622 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:48.632390 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.617667 23026 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.613847 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.619709 23202 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.638496 22033 raft_consensus.cc:606] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Received request to transfer leadership
I20251023 02:04:48.636435 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.639501 22106 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251023 02:04:48.556321 22028 rpcz_store.cc:276] 1023 02:04:47.432523 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.224955 (+792432us) service_pool.cc:225] Handling call
1023 02:04:48.487746 (+262791us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:48.619247 22102 raft_consensus.cc:1275] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:48.559864 23216 consensus_queue.cc:1048] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20251023 02:04:48.657121 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:48.645556 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.659584 22107 raft_consensus.cc:2393] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.645756 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.636229 22104 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:48.650748 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.638358 23176 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:48.637135 23047 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.630455 23253 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:48.638643 23215 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.631760 23220 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.648622 23202 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.653004 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.693365 22106 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:48.637962 23186 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.561859 23071 consensus_queue.cc:1059] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
W20251023 02:04:48.708786 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 82116.2us
I20251023 02:04:48.656558 23026 consensus_queue.cc:1048] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20251023 02:04:48.674371 23237 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:48.654912 23205 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.678086 22103 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.426705 21956 rpcz_store.cc:276] 1023 02:04:46.769878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:47.375251 (+605373us) service_pool.cc:225] Handling call
1023 02:04:47.380801 (+ 5550us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:47.485727 (+104926us) raft_consensus.cc:1534] Early marking committed up to index 5
1023 02:04:47.579254 (+ 93527us) log.cc:844] Serialized 14 byte log entry
1023 02:04:47.946744 (+367490us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:47.964614 (+ 17870us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:04:47.964677 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:48.238578 (+273901us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:04:48.332923 (+ 94345us) spinlock_profiling.cc:243] Waited 98.9 ms on lock 0x7b54000be5d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:48.332985 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:48.333103 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17560,"prepare.queue_time_us":381532,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":54183,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"spinlock_wait_cycles":230632960,"thread_start_us":270094,"threads_started":2,"wal-append.queue_time_us":472543}
I20251023 02:04:48.613397 23221 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.727702 22035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1072) took 1295 ms. Trace:
I20251023 02:04:48.665066 21928 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:48.572042 23082 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.667263 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.701166 23202 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.632490 23078 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:48.651540 23092 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:48.636073 23214 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:48.680315 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:48.680580 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 66206.9us
I20251023 02:04:48.651057 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.660122 23056 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:48.702778 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.727834 23240 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:48.703833 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.728947 23120 raft_consensus.cc:697] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:48.703842 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.705190 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:48.702986 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.671614 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:48.715291 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.774037 22106 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:48.774995 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.776352 23221 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.716015 22002 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:48.715917 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:48.714406 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.777449 22621 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.742947 23214 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:48.777815 22032 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.778846 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.679329 23168 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.636118 22587 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:48.782893 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 160950us
W20251023 02:04:48.729660 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 64723.5us
I20251023 02:04:48.757871 23082 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.655751 23178 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.717402 23047 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:48.731237 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.670904 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.732297 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.692776 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:48.785997 22106 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:48.785225 23120 consensus_queue.cc:237] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.786742 21960 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:48.758327 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.787477 22621 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:48.787647 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.788179 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.788751 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.788964 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.699716 23253 raft_consensus.cc:697] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:48.785171 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.711414 23186 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:04:48.735921 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 63729.3us
I20251023 02:04:48.663760 23251 consensus_queue.cc:1048] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20251023 02:04:48.708112 23102 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:48.745910 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.703683 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.798884 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.747403 21957 raft_consensus.cc:2368] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:48.799482 22107 raft_consensus.cc:1275] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:48.704144 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.800472 21962 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:48.737516 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:48.749620 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.807230 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.749672 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.752511 23119 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.753469 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:48.754057 23246 log.cc:927] Time spent T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20251023 02:04:48.735469 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.743760 23216 consensus_queue.cc:1048] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20251023 02:04:48.807370 22587 leader_election.cc:400] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3.
I20251023 02:04:48.757220 23263 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:48.815660 22028 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.738570 23070 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:48.758036 23166 raft_consensus.cc:993] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: : Instructing follower 1ab8b0b58147446d84d901b4ed2ff39e to start an election
I20251023 02:04:48.773195 23220 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:48.788821 21954 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:48.738778 23239 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:48.794919 21959 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:48.794145 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.728235 22035 rpcz_store.cc:276] 1023 02:04:47.432185 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.391321 (+959136us) service_pool.cc:225] Handling call
1023 02:04:48.391594 (+ 273us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:48.391786 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:04:48.621795 (+230009us) log.cc:844] Serialized 14 byte log entry
1023 02:04:48.727365 (+105570us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:48.727415 (+ 50us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:48.727463 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:48.727529 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:48.727646 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70772,"prepare.queue_time_us":307571,"thread_start_us":246507,"threads_started":2}
W20251023 02:04:48.795246 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 79326us
I20251023 02:04:48.899587 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.899724 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.789369 21960 raft_consensus.cc:2376] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3. Re-sending same reply.
W20251023 02:04:48.795593 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 59661.2us
I20251023 02:04:48.796023 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.796725 22075 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:48.801522 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.807180 23102 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.805856 21928 leader_election.cc:400] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3.
W20251023 02:04:48.905247 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 99601.8us
I20251023 02:04:48.808125 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.808188 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:48.811996 22103 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:48.812294 23215 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.793949 23240 raft_consensus.cc:697] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:48.813079 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.806926 23072 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:48.814527 21961 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:48.753242 23165 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.815750 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:48.813540 23196 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.818817 23186 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:48.797675 23092 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:48.755640 23205 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.756115 21953 raft_consensus.cc:2393] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:48.935079 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:48.739257 23078 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.943040 22029 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.943949 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:48.944474 22029 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:48.813488 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.804833 23087 consensus_queue.cc:1048] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
W20251023 02:04:48.892045 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 84233.7us
I20251023 02:04:48.804613 23047 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.829666 22587 leader_election.cc:403] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:04:48.813918 23071 consensus_queue.cc:1048] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20251023 02:04:48.891463 22031 raft_consensus.cc:2393] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.803349 22000 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:04:48.902164 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 105518us
I20251023 02:04:48.905333 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:48.993775 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 190549us
I20251023 02:04:48.910681 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:48.994549 22034 raft_consensus.cc:2376] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3. Re-sending same reply.
I20251023 02:04:48.910326 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.995450 21959 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:48.996218 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:48.996835 21959 raft_consensus.cc:2393] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:04:48.996978 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 213487us
I20251023 02:04:48.910972 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.979997 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.998299 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.998466 22587 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:48.998766 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.999370 21956 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:48.999581 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:49.000151 23237 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3.
I20251023 02:04:49.000203 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.000454 21960 raft_consensus.cc:2368] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3.
I20251023 02:04:49.000923 23237 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:48.908640 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:49.001741 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.002216 23245 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.002375 22105 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.002799 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:49.003824 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.004460 22035 raft_consensus.cc:2393] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.836098 23263 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:49.005755 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.006276 23171 consensus_queue.cc:1059] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:49.006897 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.007293 23263 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.893908 23176 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.989842 23175 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:49.008904 23175 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.009238 23176 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:48.909117 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.924468 23026 raft_consensus.cc:3055] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:49.014869 23026 raft_consensus.cc:740] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:49.014973 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:49.013872 23071 raft_consensus.cc:3055] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:49.015661 23071 raft_consensus.cc:740] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:49.015790 22104 raft_consensus.cc:2393] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:49.015828 23026 consensus_queue.cc:260] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:49.017235 23026 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:48.924855 23201 raft_consensus.cc:1081] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Signalling peer 1ab8b0b58147446d84d901b4ed2ff39e to start an election
I20251023 02:04:48.925313 23271 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:48.814857 23253 consensus_queue.cc:237] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.018039 23271 raft_consensus.cc:697] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:49.019202 23175 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:49.019001 23271 consensus_queue.cc:237] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:48.926102 23102 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.926772 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.020990 21955 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:48.926288 22109 raft_consensus.cc:2368] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3.
I20251023 02:04:48.927126 23215 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:49.021908 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:49.023000 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.023168 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1081) took 1585 ms. Trace:
I20251023 02:04:49.023567 22031 rpcz_store.cc:276] 1023 02:04:47.437479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.373145 (+935666us) service_pool.cc:225] Handling call
1023 02:04:49.017524 (+644379us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54002a76d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:49.023114 (+ 5590us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":842303744}
I20251023 02:04:49.024034 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.024745 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:48.928122 21958 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:48.902029 23238 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.901793 23120 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:48.920446 22618 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.030138 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:48.945691 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:48.936751 23072 raft_consensus.cc:697] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:48.960078 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:49.041862 23072 consensus_queue.cc:237] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.001739 23060 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:49.043181 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:49.043427 22620 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507"
dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
from {username='slave'} at 127.0.0.1:34820
I20251023 02:04:49.044059 22620 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:49.044453 22620 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.044169 23120 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.963639 23216 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:48.960146 23220 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:48.945041 23270 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:48.965420 23273 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:48.966460 23178 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.968789 23165 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:48.904393 23239 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.065835 23273 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.972369 23240 consensus_queue.cc:237] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.069759 23273 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:48.929173 23082 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:49.071269 23071 consensus_queue.cc:260] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.071702 23082 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.072530 23270 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.974092 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:49.038353 21954 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:49.074405 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.074581 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.075537 22030 raft_consensus.cc:2393] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:49.075778 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:48.975951 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.981732 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.963770 23202 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.987186 23056 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:49.095917 23056 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:48.990010 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:48.910934 23214 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.009985 22110 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:49.011173 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:49.011255 22000 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:48.997469 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:49.098806 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.099038 21959 raft_consensus.cc:2393] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:49.099243 23214 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:49.099473 23202 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:49.099910 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:49.099478 22034 raft_consensus.cc:2393] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:49.062065 23216 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.101097 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.005172 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:49.101351 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.102367 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.102535 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.103399 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:49.103864 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.012364 22002 leader_election.cc:400] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3.
I20251023 02:04:48.994025 21962 raft_consensus.cc:2376] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3. Re-sending same reply.
I20251023 02:04:48.991854 23186 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:49.020412 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:49.120368 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.023008 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:49.121662 22033 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:49.001392 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:49.120177 23178 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:49.122951 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.123168 21956 raft_consensus.cc:2393] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:49.123099 23178 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:49.123628 23178 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:49.038311 21957 raft_consensus.cc:2376] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3. Re-sending same reply.
I20251023 02:04:49.124022 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.124742 22031 raft_consensus.cc:2393] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:49.040282 21925 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:49.043197 23060 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:49.037343 22104 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:49.124024 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:48.996829 23047 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:49.127116 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:49.127447 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:49.127679 21956 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.125787 23060 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:49.054590 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:49.129043 21958 raft_consensus.cc:2393] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:49.129515 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.129788 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.130117 22031 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.130540 21958 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:49.068341 23220 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.064774 22618 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:49.025228 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.144441 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:49.145542 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.146126 21955 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:49.068930 23239 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:48.975265 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.077273 21961 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:49.150502 22107 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:49.151836 21961 raft_consensus.cc:1240] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251023 02:04:49.077816 22036 raft_consensus.cc:2393] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:49.095266 22103 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:49.100389 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.157491 21953 raft_consensus.cc:2393] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:49.157788 23237 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:04:49.001261 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.104773 22002 leader_election.cc:403] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20251023 02:04:49.160729 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 148523us
I20251023 02:04:49.013309 22621 raft_consensus.cc:1240] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
W20251023 02:04:49.104807 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:49.099946 22108 raft_consensus.cc:2393] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:49.105804 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.121827 22035 raft_consensus.cc:2393] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:48.897270 23070 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:49.166007 23070 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.166025 21962 raft_consensus.cc:2393] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:49.153360 21961 raft_consensus.cc:1275] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:49.124603 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.038616 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:49.125710 22034 raft_consensus.cc:2393] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:49.057324 22102 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251023 02:04:49.138554 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:49.142838 23171 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:49.172971 23171 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:49.173384 23171 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.181207 22587 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.184947 23237 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:04:49.205708 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 165548us
I20251023 02:04:49.206935 21925 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:49.208443 21925 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.209657 23168 raft_consensus.cc:2749] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.210342 23220 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:49.211536 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 200404us
I20251023 02:04:49.212426 23047 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:49.213824 22000 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.213722 23165 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:04:49.214632 23201 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:04:49.217679 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 54917.5us
I20251023 02:04:49.217890 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:49.218612 21859 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56872 (request call id 211) took 2281 ms. Trace:
I20251023 02:04:49.219187 22030 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:49.219305 21856 catalog_manager.cc:5649] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 2, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 2 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:04:49.221205 21856 catalog_manager.cc:5649] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 3, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:49.222934 23270 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:49.225368 22620 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:49.226675 23071 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.228309 22105 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
W20251023 02:04:49.229633 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 48527.4us
I20251023 02:04:49.219136 22002 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:49.230669 23082 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:49.230551 23237 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:49.207201 23120 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:49.231904 22033 raft_consensus.cc:1240] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251023 02:04:49.217501 22000 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.236697 23165 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.237138 21955 raft_consensus.cc:1240] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251023 02:04:49.238183 22033 raft_consensus.cc:1275] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:49.238247 22000 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.219017 21859 rpcz_store.cc:276] 1023 02:04:47.024122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:47.031232 (+ 7110us) service_pool.cc:225] Handling call
1023 02:04:49.218551 (+2187319us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:04:48.239928 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:04:48.277972 (+ 38044us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:04:48.278020 (+ 48us) write_op.cc:435] Acquired schema lock
1023 02:04:48.278047 (+ 27us) tablet.cc:660] Decoding operations
1023 02:04:48.335498 (+ 57451us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:04:48.335575 (+ 77us) write_op.cc:641] Partition lock acquired for write op
1023 02:04:48.335604 (+ 29us) tablet.cc:683] Acquiring locks for 4 operations
1023 02:04:48.335966 (+ 362us) tablet.cc:699] Row locks acquired
1023 02:04:48.335986 (+ 20us) write_op.cc:260] PREPARE: finished
1023 02:04:48.336110 (+ 124us) write_op.cc:270] Start()
1023 02:04:48.336223 (+ 113us) write_op.cc:276] Timestamp: P: 1761185088336074 usec, L: 0
1023 02:04:48.336243 (+ 20us) op_driver.cc:348] REPLICATION: starting
1023 02:04:48.338696 (+ 2453us) log.cc:844] Serialized 1681 byte log entry
1023 02:04:48.899186 (+560490us) op_driver.cc:464] REPLICATION: finished
1023 02:04:48.988137 (+ 88951us) write_op.cc:301] APPLY: starting
1023 02:04:48.988217 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
1023 02:04:48.988356 (+ 139us) tablet.cc:1368] finished BulkCheckPresence
1023 02:04:48.988466 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:04:48.989041 (+ 575us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:04:48.989211 (+ 170us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
1023 02:04:48.989266 (+ 55us) write_op.cc:312] APPLY: finished
1023 02:04:48.989543 (+ 277us) log.cc:844] Serialized 40 byte log entry
1023 02:04:49.013649 (+ 24106us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:49.036642 (+ 22993us) write_op.cc:454] Released schema lock
1023 02:04:49.046860 (+ 10218us) write_op.cc:341] FINISH: Updating metrics
1023 02:04:49.060562 (+ 13702us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:49.060609 (+ 47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":253951,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":75442,"apply.run_cpu_time_us":2385,"apply.run_wall_time_us":72334,"num_ops":4,"prepare.queue_time_us":202437,"prepare.run_cpu_time_us":92338,"prepare.run_wall_time_us":364725,"raft.queue_time_us":175365,"raft.run_cpu_time_us":50086,"raft.run_wall_time_us":162204,"replication_time_us":500000,"thread_start_us":542747,"threads_started":4,"wal-append.queue_time_us":294492}]]}
I20251023 02:04:49.207741 23078 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.222880 21856 catalog_manager.cc:5649] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 2, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 2 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:49.244885 22000 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:49.209679 21925 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.248871 21925 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.249094 22000 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.250177 23252 raft_consensus.cc:2749] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.250356 23216 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:49.252698 21928 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:49.249182 21856 catalog_manager.cc:5649] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:49.221467 22030 raft_consensus.cc:1275] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:49.255525 22000 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:49.255395 21856 catalog_manager.cc:5649] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:49.258298 22000 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.258606 21856 catalog_manager.cc:5649] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 3, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:49.259258 23166 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.260800 22000 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.265830 22111 raft_consensus.cc:1240] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:49.274859 21925 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:49.247757 23047 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.251667 23082 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.254437 23090 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:49.248262 22028 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:49.269763 23102 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.256660 23168 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:49.299032 22000 leader_election.cc:400] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3.
I20251023 02:04:49.270433 23087 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:49.245769 23165 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20251023 02:04:49.290318 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:49.270771 23005 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:49.254626 23201 consensus_queue.cc:1048] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20251023 02:04:49.279742 23166 raft_consensus.cc:2749] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.277117 23175 consensus_queue.cc:1048] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
W20251023 02:04:49.319413 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 241077us
I20251023 02:04:49.270591 23178 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.328656 22590 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:49.291946 23072 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:49.282807 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.301946 23056 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:49.301560 23090 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:49.311019 22000 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.264689 23170 consensus_queue.cc:1059] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20251023 02:04:49.320008 23082 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:49.334993 23070 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:49.335875 21956 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:49.336098 22621 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:49.337859 22030 raft_consensus.cc:1240] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:49.338661 22000 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.321038 23120 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:49.351663 23120 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.352308 22037 raft_consensus.cc:1240] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:49.352483 23087 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:49.353065 23072 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:49.353749 23120 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:49.344951 23267 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.338776 23242 raft_consensus.cc:2804] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:49.358103 23242 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:49.358476 23242 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.357300 23072 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:49.345907 22620 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:49.340756 23082 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:49.361619 23082 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.363041 23078 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:49.363307 21959 raft_consensus.cc:2410] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:04:49.364431 23070 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:49.367517 21957 raft_consensus.cc:2393] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:49.350839 23102 raft_consensus.cc:3055] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:49.332370 23005 raft_consensus.cc:697] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:49.405967 21961 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:49.339649 23090 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:49.408221 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.408834 22030 raft_consensus.cc:3055] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:49.409166 22030 raft_consensus.cc:740] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:49.409516 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:49.353034 23087 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:49.410429 22002 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.341835 23196 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:49.326608 23205 consensus_queue.cc:1048] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20251023 02:04:49.416014 22000 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:49.420217 21953 raft_consensus.cc:2410] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:04:49.414574 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.415263 22031 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:49.411535 23171 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:49.382081 23102 raft_consensus.cc:740] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:49.409899 22030 consensus_queue.cc:260] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:49.426620 22002 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:49.436115 22030 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.407130 23092 consensus_queue.cc:1059] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20251023 02:04:49.461721 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:49.459947 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:49.459060 22109 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251023 02:04:49.454618 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:04:49.508610 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 233852us
I20251023 02:04:49.431072 23227 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.509879 21958 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:49.509533 21925 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.488278 22077 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:49.479360 22029 consensus_meta.cc:287] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.534s user 0.012s sys 0.000s
I20251023 02:04:49.436856 23240 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:49.446246 23251 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:49.522152 21925 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:49.524024 21925 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:49.530071 21928 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:49.531230 21925 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.531627 22036 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251023 02:04:49.532872 22077 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:49.510656 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 158373us
I20251023 02:04:49.483916 23166 consensus_queue.cc:1059] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20251023 02:04:49.515494 23253 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.574301 22077 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:49.535513 23070 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:49.517618 23078 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:49.445511 23060 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.535351 23238 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:49.516321 22029 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
W20251023 02:04:49.576297 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 45205.2us
I20251023 02:04:49.517202 23240 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.603948 22077 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:49.539939 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.541842 23251 raft_consensus.cc:697] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:04:49.622879 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 174065us
W20251023 02:04:49.623457 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 196984us
I20251023 02:04:49.583293 23170 raft_consensus.cc:2749] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.592510 23168 raft_consensus.cc:2804] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:49.590847 23026 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251023 02:04:49.613740 23254 log.cc:927] Time spent T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.072s user 0.000s sys 0.009s
I20251023 02:04:49.619084 23239 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3.
I20251023 02:04:49.635702 22002 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.539526 23205 consensus_queue.cc:1059] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20251023 02:04:49.636865 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.610000 23245 raft_consensus.cc:3055] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:49.599872 23070 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:49.603322 23270 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.517556 23005 consensus_queue.cc:237] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:49.613741 23290 log.cc:927] Time spent T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20251023 02:04:49.626863 23120 consensus_queue.cc:1059] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20251023 02:04:49.633774 23214 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:49.654649 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:49.635423 22103 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.638335 23178 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:49.538578 23102 consensus_queue.cc:260] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.659619 22030 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:49.653772 23175 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:49.659087 23168 raft_consensus.cc:697] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:49.667829 23070 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:49.663484 23245 raft_consensus.cc:740] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:49.651165 23170 consensus_queue.cc:1048] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
W20251023 02:04:49.718571 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 188589us
W20251023 02:04:49.727137 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 111657us
I20251023 02:04:49.727854 21925 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.729159 21925 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.730779 21925 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.736279 21925 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:49.714421 23294 log.cc:927] Time spent T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20251023 02:04:49.714103 23214 raft_consensus.cc:697] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
W20251023 02:04:49.777209 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 154388us
I20251023 02:04:49.715549 23171 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:49.734213 23102 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:49.735527 23196 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:49.732072 23071 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.736037 23205 raft_consensus.cc:2749] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:49.714928 23175 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.789462 23047 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.792636 23252 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:49.800830 23078 raft_consensus.cc:697] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:49.819340 23026 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.825839 23196 raft_consensus.cc:697] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:49.737015 23251 consensus_queue.cc:237] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.838704 23252 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:49.855769 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 119618us
I20251023 02:04:49.858420 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.859611 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.879594 22076 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:49.843137 23060 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:49.842186 23304 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:49.843305 21954 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:49.852042 23240 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:49.826208 23245 consensus_queue.cc:260] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:49.873999 23082 consensus_meta.cc:287] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.512s user 0.009s sys 0.000s
I20251023 02:04:49.885845 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.895730 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.896703 22108 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:49.897635 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.898954 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:49.899823 22108 raft_consensus.cc:2393] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:49.900679 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:49.857034 23056 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251023 02:04:49.925664 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:49.931939 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:49.974574 23072 consensus_meta.cc:287] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.615s user 0.008s sys 0.000s
I20251023 02:04:49.975039 23072 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:49.975304 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:49.976800 22589 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:49.977089 23072 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:49.977166 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:49.979048 23242 consensus_meta.cc:287] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.620s user 0.009s sys 0.001s
I20251023 02:04:49.979450 22102 raft_consensus.cc:1275] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251023 02:04:49.980475 23090 consensus_meta.cc:287] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.550s user 0.004s sys 0.004s
I20251023 02:04:49.980871 23090 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:49.902917 23078 consensus_queue.cc:237] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:49.983353 23303 log.cc:927] Time spent T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.160s user 0.000s sys 0.000s
I20251023 02:04:49.984405 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1105) took 1025 ms. Trace:
W20251023 02:04:49.986017 23301 log.cc:927] Time spent T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.157s user 0.000s sys 0.000s
I20251023 02:04:49.986099 23092 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:49.987174 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 57) took 1059 ms. Trace:
I20251023 02:04:49.988456 23216 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:49.987685 21955 rpcz_store.cc:276] 1023 02:04:48.928098 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.288029 (+359931us) service_pool.cc:225] Handling call
1023 02:04:49.326232 (+ 38203us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:49.382140 (+ 55908us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:49.406355 (+ 24215us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:49.509251 (+102896us) log.cc:844] Serialized 26 byte log entry
1023 02:04:49.642536 (+133285us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:49.642601 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:49.650688 (+ 8087us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:49.986882 (+336194us) raft_consensus.cc:1685] finished
1023 02:04:49.986909 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:49.987101 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34423,"raft.queue_time_us":38580,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":17471,"thread_start_us":109404,"threads_started":1,"wal-append.queue_time_us":267796}
W20251023 02:04:49.990363 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:49.989542 23298 log.cc:927] Time spent T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.259s user 0.001s sys 0.000s
I20251023 02:04:49.843116 23168 consensus_queue.cc:237] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:49.907326 23082 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:49.992955 23082 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:49.993432 22111 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:50.002847 22028 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:49.984720 21959 rpcz_store.cc:276] 1023 02:04:48.958713 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.365634 (+406921us) service_pool.cc:225] Handling call
1023 02:04:49.365933 (+ 299us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:49.433899 (+ 67966us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:49.433926 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:49.434362 (+ 436us) log.cc:844] Serialized 26 byte log entry
1023 02:04:49.626660 (+192298us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:49.626713 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:49.626914 (+ 201us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:49.984223 (+357309us) raft_consensus.cc:1685] finished
1023 02:04:49.984241 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:49.984353 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76749,"raft.queue_time_us":68454,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":43919,"thread_start_us":112449,"threads_started":1,"wal-append.queue_time_us":334654}
W20251023 02:04:50.006743 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 127288us
I20251023 02:04:50.020665 21962 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20251023 02:04:50.010296 22076 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:50.023861 22076 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:50.025210 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:49.911126 23245 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:04:49.987243 23302 log.cc:927] Time spent T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.195s user 0.001s sys 0.000s
I20251023 02:04:50.052176 21959 raft_consensus.cc:1275] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:50.052580 22076 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:50.046985 22000 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.006s sys 0.020s
I20251023 02:04:50.082723 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:50.083566 21959 raft_consensus.cc:2368] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:50.084558 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:50.089811 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:50.082906 22000 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.569s user 0.006s sys 0.020s
I20251023 02:04:50.051657 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.096155 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.008301 23240 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:50.097754 21925 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:50.100935 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.111068 21925 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:50.073827 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:50.112275 21925 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:50.112511 23078 raft_consensus.cc:2749] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:04:50.114526 21925 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:50.114813 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:50.049268 23090 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:50.118218 21925 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:50.096634 23087 consensus_meta.cc:287] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.663s user 0.010s sys 0.000s
I20251023 02:04:50.122113 21925 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:50.120626 22000 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:50.087831 23216 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:50.125921 22000 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:50.119925 22106 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.029s sys 0.030s
I20251023 02:04:49.894433 23214 consensus_queue.cc:237] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:50.127445 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:50.127830 22106 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s user 0.030s sys 0.030s
I20251023 02:04:50.137509 22033 raft_consensus.cc:1275] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:50.136381 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:50.142534 22033 raft_consensus.cc:1275] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:50.144734 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:50.146473 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:50.145758 22033 raft_consensus.cc:2368] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:50.148932 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:50.149847 22035 raft_consensus.cc:2368] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:50.151417 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:49.979471 23242 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:50.158601 21925 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:50.097002 23316 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:50.161914 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:50.116505 23205 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:50.160164 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:50.117167 23171 raft_consensus.cc:2749] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:50.168314 21925 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:50.118921 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.121004 23170 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:50.171617 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.131258 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:50.171216 22035 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:50.166603 23071 raft_consensus.cc:2749] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:50.021083 23196 consensus_queue.cc:237] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:50.164728 22104 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.035s sys 0.060s
I20251023 02:04:50.130934 23175 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.165421 22103 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:50.131319 23312 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:50.213001 23297 raft_consensus.cc:3055] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:50.213413 23297 raft_consensus.cc:740] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:50.127296 23315 raft_consensus.cc:2749] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:50.214159 23297 consensus_queue.cc:260] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.215176 23297 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:04:50.182356 23283 log.cc:927] Time spent T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.668s user 0.000s sys 0.004s
I20251023 02:04:50.243213 22108 raft_consensus.cc:3055] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:50.254319 22108 raft_consensus.cc:740] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:50.216532 21953 raft_consensus.cc:2368] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
W20251023 02:04:50.271096 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 220200us
I20251023 02:04:50.269974 22108 consensus_queue.cc:260] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.174028 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:50.178684 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:50.276158 22108 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:50.194283 23252 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.194962 21955 raft_consensus.cc:2393] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:50.196530 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:50.184595 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 207901us
I20251023 02:04:50.164916 23216 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:50.200387 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:50.117496 23240 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:50.206344 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:50.285728 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:50.285749 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:50.287184 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:50.173121 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:50.133734 23087 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:50.204821 22104 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s user 0.035s sys 0.060s
I20251023 02:04:50.238804 22077 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:50.292744 23216 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:04:50.293169 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 54476.3us
I20251023 02:04:50.294035 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1090) took 1378 ms. Trace:
I20251023 02:04:50.283986 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:50.295228 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:50.180982 23205 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:50.244052 22000 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:50.250276 23242 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.250578 23090 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:50.251353 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:50.118582 23070 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:50.194190 23310 log.cc:927] Time spent T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20251023 02:04:50.279414 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:50.288422 22030 raft_consensus.cc:2393] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:04:50.297044 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 53105.9us
I20251023 02:04:50.308918 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:50.297631 22035 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:50.294396 21958 rpcz_store.cc:276] 1023 02:04:48.915345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.131022 (+215677us) service_pool.cc:225] Handling call
1023 02:04:50.293855 (+1162833us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b5400106b54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:50.293981 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":774916352}
I20251023 02:04:50.278537 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:50.311055 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:50.304275 23253 consensus_meta.cc:287] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.696s user 0.011s sys 0.000s
I20251023 02:04:50.312789 22000 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:50.305975 22031 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:50.298105 23216 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:50.316082 22035 raft_consensus.cc:1240] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251023 02:04:50.297688 23220 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:50.318168 22110 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251023 02:04:50.292744 23240 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:50.321396 22000 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:50.323434 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1121) took 1309 ms. Trace:
I20251023 02:04:50.323827 22036 rpcz_store.cc:276] 1023 02:04:49.014096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.320910 (+306814us) service_pool.cc:225] Handling call
1023 02:04:49.432474 (+111564us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:49.532042 (+ 99568us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:49.858580 (+326538us) spinlock_profiling.cc:243] Waited 98.9 ms on lock 0x7b540040d8d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:50.323259 (+464679us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b540040d8d0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:50.323381 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":483175936}
I20251023 02:04:50.331302 22030 raft_consensus.cc:1240] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:50.331856 22000 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:50.331431 22035 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:50.334700 22077 leader_election.cc:400] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:50.335052 22077 leader_election.cc:403] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251023 02:04:50.320171 21956 raft_consensus.cc:1240] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:50.317272 23253 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:50.298166 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:50.344508 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.346854 22033 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
W20251023 02:04:50.348382 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:50.348836 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:50.344213 22108 raft_consensus.cc:1275] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251023 02:04:50.307540 23316 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20251023 02:04:50.310391 23090 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:50.322122 23252 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20251023 02:04:50.356513 22108 pending_rounds.cc:85] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 1
I20251023 02:04:50.330379 23060 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:50.333335 23216 raft_consensus.cc:697] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:50.334189 23102 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:50.347782 23120 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:50.345141 22077 leader_election.cc:400] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:50.422470 22077 leader_election.cc:403] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20251023 02:04:50.423102 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 78116.4us
W20251023 02:04:50.406472 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:50.408668 22108 pending_rounds.cc:107] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251023 02:04:50.424324 22108 raft_consensus.cc:2889] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:04:50.321360 23323 consensus_queue.cc:1059] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20251023 02:04:50.346892 23005 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:50.333380 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:50.347365 23308 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:50.401291 23315 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:50.427634 22077 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:50.429530 23324 log.cc:927] Time spent T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
I20251023 02:04:50.432647 23270 raft_consensus.cc:3055] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:50.432825 22620 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20251023 02:04:50.454288 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 401818us
I20251023 02:04:50.455382 22076 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:50.456171 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:50.456274 23271 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:50.456749 23271 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:50.456589 22076 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:50.457219 23271 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:50.457796 23319 raft_consensus.cc:2749] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:04:49.445905 23023 consensus_queue.cc:1175] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" responder_term: 3 status { last_received { term: 2 index: 4 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251023 02:04:50.435048 23087 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.335578 23240 consensus_queue.cc:1048] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
I20251023 02:04:50.436911 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1101) took 1498 ms. Trace:
I20251023 02:04:50.351398 23311 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:50.440817 23078 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:50.460055 22076 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:50.446475 23328 log.cc:927] Time spent T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20251023 02:04:50.468586 23319 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:50.468479 23165 consensus_queue.cc:1048] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:50.463475 21962 rpcz_store.cc:276] 1023 02:04:48.938435 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.350548 (+412113us) service_pool.cc:225] Handling call
1023 02:04:49.351026 (+ 478us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:50.022845 (+671819us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:50.436484 (+413639us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b540030f7d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:50.436808 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1501381888}
I20251023 02:04:50.469763 22108 raft_consensus.cc:1240] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251023 02:04:50.470335 21955 raft_consensus.cc:2368] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:50.407369 23332 consensus_queue.cc:1048] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.112s
I20251023 02:04:50.470288 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.471170 21962 raft_consensus.cc:2393] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:50.471318 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.469769 22076 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:50.471951 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.472479 23332 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:50.473049 23332 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:50.473404 23332 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:50.462710 23251 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20251023 02:04:50.474138 23311 raft_consensus.cc:697] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:50.475051 23311 consensus_queue.cc:237] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.446823 23090 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:50.478463 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:50.479079 22621 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:50.461095 23315 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.473064 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.482275 23070 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.459012 23270 raft_consensus.cc:740] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:50.483476 23270 consensus_queue.cc:260] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.484481 23270 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:50.485188 23220 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:50.485684 23220 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:50.486093 23325 log.cc:927] Time spent T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
I20251023 02:04:50.487177 22104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1110) took 1023 ms. Trace:
I20251023 02:04:50.487587 22104 rpcz_store.cc:276] 1023 02:04:49.463801 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.479491 (+ 15690us) service_pool.cc:225] Handling call
1023 02:04:49.534704 (+ 55213us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:49.621735 (+ 87031us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:49.622538 (+ 803us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:49.662076 (+ 39538us) log.cc:844] Serialized 26 byte log entry
1023 02:04:50.292565 (+630489us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:50.292608 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:50.292687 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:50.486962 (+194275us) raft_consensus.cc:1685] finished
1023 02:04:50.486988 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:50.487118 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4530,"raft.queue_time_us":32944,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":11697,"thread_start_us":630106,"threads_started":1,"wal-append.queue_time_us":636987}
W20251023 02:04:50.490379 23320 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
I20251023 02:04:50.491171 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.488052 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:50.491487 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1111) took 1011 ms. Trace:
I20251023 02:04:50.426162 23102 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.492305 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:50.492962 21959 raft_consensus.cc:2393] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:50.493960 23102 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.491856 22106 rpcz_store.cc:276] 1023 02:04:49.479753 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.481224 (+ 1471us) service_pool.cc:225] Handling call
1023 02:04:49.484740 (+ 3516us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:49.485136 (+ 396us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:49.485183 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:49.513617 (+ 28434us) log.cc:844] Serialized 548 byte log entry
1023 02:04:50.128343 (+614726us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:04:50.128382 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:50.128443 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:50.491311 (+362868us) raft_consensus.cc:1685] finished
1023 02:04:50.491330 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:50.491437 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":614468,"threads_started":1,"wal-append.queue_time_us":764990}
I20251023 02:04:50.481287 23315 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.472460 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.496048 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.461957 23316 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:50.498554 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:50.497200 22076 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:50.505199 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:50.507361 22076 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:50.507789 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:50.508025 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 339826us
I20251023 02:04:50.510318 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.513967 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:50.508109 23239 consensus_queue.cc:1059] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20251023 02:04:50.515934 22103 raft_consensus.cc:1275] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:50.501808 23078 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:04:50.517969 23289 log.cc:927] Time spent T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 1.127s user 0.004s sys 0.002s
I20251023 02:04:50.518646 22077 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:50.519181 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 49) took 1589 ms. Trace:
I20251023 02:04:50.519582 22034 rpcz_store.cc:276] 1023 02:04:48.929356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.211718 (+282362us) service_pool.cc:225] Handling call
1023 02:04:49.212920 (+ 1202us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:49.213382 (+ 462us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:49.213410 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:49.213847 (+ 437us) log.cc:844] Serialized 26 byte log entry
1023 02:04:49.273606 (+ 59759us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:49.273645 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:49.284032 (+ 10387us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:50.517363 (+1233331us) spinlock_profiling.cc:243] Waited 5.35 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:04:50.518898 (+ 1535us) raft_consensus.cc:1685] finished
1023 02:04:50.518919 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:50.519122 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31,"raft.queue_time_us":71,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"spinlock_wait_cycles":12296064,"thread_start_us":59308,"threads_started":1,"wal-append.queue_time_us":155579}
I20251023 02:04:50.521003 22106 raft_consensus.cc:1275] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:50.522522 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.523447 21958 raft_consensus.cc:2368] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:50.511184 21959 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:50.525336 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.503686 23316 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:50.528014 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:50.530437 21959 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:50.547595 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:50.548390 21959 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:50.552058 21925 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:50.554426 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:50.554759 21959 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f"
dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
mode: GRACEFUL
new_leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
from {username='slave'} at 127.0.0.1:40860
I20251023 02:04:50.555377 21959 raft_consensus.cc:606] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Received request to transfer leadership to TS 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:50.555701 21959 raft_consensus.cc:614] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:04:50.528802 23239 consensus_queue.cc:1048] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
W20251023 02:04:50.546450 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:50.558238 21958 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:04:50.554102 23201 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.033s user 0.016s sys 0.006s
I20251023 02:04:50.546516 23316 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:50.498143 23216 consensus_queue.cc:237] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.567093 21959 raft_consensus.cc:1275] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:50.558324 23078 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251023 02:04:50.564442 23201 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.050s user 0.017s sys 0.006s
I20251023 02:04:50.558465 23102 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:50.613509 21958 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:50.549223 23196 consensus_queue.cc:1048] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20251023 02:04:50.624439 21959 raft_consensus.cc:1275] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251023 02:04:50.615309 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:50.583446 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:50.627269 23332 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.646909 23242 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:50.647571 23242 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:50.622148 22000 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:50.629791 21958 raft_consensus.cc:1240] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:50.630331 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:50.630470 23334 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:50.560037 23319 consensus_queue.cc:1059] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.064s
I20251023 02:04:50.637276 22075 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:50.617961 23078 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:50.589818 23082 raft_consensus.cc:2749] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:50.628418 22618 raft_consensus.cc:1240] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:50.560076 23331 consensus_queue.cc:1059] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.076s
I20251023 02:04:50.644270 23087 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:50.669715 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:50.670550 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:50.688261 22108 raft_consensus.cc:2393] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
W20251023 02:04:50.671396 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 164140us
I20251023 02:04:50.674399 23347 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:50.674888 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:50.675179 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:50.677136 23305 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:50.689980 23248 consensus_queue.cc:1059] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:50.664667 23078 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:50.691114 23337 log.cc:927] Time spent T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20251023 02:04:50.691409 23087 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.694860 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1111) took 1393 ms. Trace:
I20251023 02:04:50.700498 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:50.700872 23245 consensus_meta.cc:287] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.613s user 0.002s sys 0.006s
I20251023 02:04:50.703598 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:38996 (request call id 651) took 1270 ms. Trace:
W20251023 02:04:50.705741 21954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36488 (request call id 58) took 1767 ms (client timeout 2040 ms). Trace:
I20251023 02:04:50.707749 23332 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:50.708091 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 51) took 1675 ms. Trace:
I20251023 02:04:50.710451 23092 raft_consensus.cc:3055] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:50.711741 23120 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:50.714541 23166 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:50.716706 23242 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:50.662276 22589 leader_election.cc:336] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.578s (SENT)
I20251023 02:04:50.721129 21955 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3"
dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
mode: GRACEFUL
new_leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
from {username='slave'} at 127.0.0.1:40860
W20251023 02:04:50.715840 23297 consensus_meta.cc:287] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.500s user 0.010s sys 0.002s
I20251023 02:04:50.640163 23271 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.970731 22037 raft_consensus.cc:2393] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:50.861035 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:50.971427 22103 raft_consensus.cc:1275] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:50.971879 23166 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.798164 23316 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.973654 22107 raft_consensus.cc:1240] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:50.974005 23166 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.974334 23336 raft_consensus.cc:3055] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:51.008982 23336 raft_consensus.cc:740] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:51.010190 23336 consensus_queue.cc:260] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.011325 23242 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.012053 23336 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:51.011864 23242 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:50.697252 23220 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.014657 22106 raft_consensus.cc:1240] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:51.014837 23242 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:51.017930 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:51.019224 22106 raft_consensus.cc:1240] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:50.941002 23296 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.020887 23296 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.027300 23296 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.861858 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:51.028891 22615 raft_consensus.cc:2393] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 2: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:51.019160 22104 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:51.029266 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.030925 22036 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:51.032469 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.033071 22031 raft_consensus.cc:2376] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3. Re-sending same reply.
I20251023 02:04:51.034014 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.034726 22031 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:51.035480 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.036505 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:50.977198 23087 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.038661 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.039249 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.039861 22104 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:50.977510 22032 rpcz_store.cc:276] 1023 02:04:49.433122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.434575 (+ 1453us) service_pool.cc:225] Handling call
1023 02:04:50.702401 (+1267826us) spinlock_profiling.cc:243] Waited 80.4 ms on lock 0x7b54002a8ad4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa02fe0 00007f9c5faa1aca 00007f9c64a4701e 00007f9c5dfee437 00007f9c5dfee3c2 00007f9c5dfee2f1 00007f9c5dfee26c 00007f9c5dfed522 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:50.703485 (+ 1084us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":184966912}
I20251023 02:04:51.032495 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:50.790189 23201 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.042037 23201 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:50.862769 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 173772us
I20251023 02:04:51.043226 22034 raft_consensus.cc:1275] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:51.044566 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.045130 22034 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:51.045441 22076 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:51.046007 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:50.919183 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:51.046541 22034 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:50.979599 21956 rpcz_store.cc:276] 1023 02:04:49.301951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.421675 (+119724us) service_pool.cc:225] Handling call
1023 02:04:49.510375 (+ 88700us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:50.346753 (+836378us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:50.694620 (+347867us) spinlock_profiling.cc:243] Waited 795 ms on lock 0x7b540017dc54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:50.694799 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1828283776}
I20251023 02:04:51.046928 23214 consensus_queue.cc:1059] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:51.047730 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.048007 23214 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:51.046555 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.049117 23201 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.049284 22031 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:50.981251 21962 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:51.050349 22031 raft_consensus.cc:1240] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:51.050689 23220 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:51.051394 23214 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:51.039268 22037 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:51.052819 22076 leader_election.cc:400] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:50.861927 22624 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251023 02:04:51.053387 23253 consensus_queue.cc:1059] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:51.054085 22076 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:50.981969 21954 rpcz_store.cc:269] 1023 02:04:48.937953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.075496 (+137543us) service_pool.cc:225] Handling call
1023 02:04:50.705473 (+1629977us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b5400108454. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:50.705666 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":665900800}
I20251023 02:04:50.982630 23305 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.054808 23216 consensus_queue.cc:1048] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:50.906116 22033 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20251023 02:04:51.056146 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:51.056455 23216 raft_consensus.cc:2749] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:51.056185 23305 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.733394 23239 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:50.928939 23221 raft_consensus.cc:3055] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:50.869271 23056 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:51.063104 23221 raft_consensus.cc:740] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:51.063277 23056 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.063930 23221 consensus_queue.cc:260] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.064857 23221 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:51.065230 23056 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:50.935052 23315 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:50.888702 23196 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:50.769655 22622 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251023 02:04:50.895732 22035 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:50.732329 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.066040 23196 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.816660 23319 raft_consensus.cc:3055] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:51.067373 23319 raft_consensus.cc:740] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:51.068029 23196 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:50.743232 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:51.068361 23319 consensus_queue.cc:260] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.768819 22619 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251023 02:04:50.733278 22108 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251023 02:04:50.988852 23297 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.071496 23297 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:50.986835 22107 raft_consensus.cc:1240] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251023 02:04:50.869643 23270 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:51.073324 23297 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.073845 23270 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.074338 23319 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.075713 23270 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:50.757427 22029 raft_consensus.cc:1240] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:51.077169 22029 raft_consensus.cc:1240] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:51.078887 23297 consensus_queue.cc:1059] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:51.079304 22029 raft_consensus.cc:1240] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20251023 02:04:51.080067 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.080489 22029 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:51.081107 22104 raft_consensus.cc:2393] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:50.820293 21858 catalog_manager.cc:5649] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 3, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 3 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:04:51.065124 23239 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.083716 23239 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.763371 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.087080 21961 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:51.087518 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36488 (request call id 62) took 1007 ms. Trace:
I20251023 02:04:51.087812 21961 rpcz_store.cc:276] 1023 02:04:50.079952 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:50.117290 (+ 37338us) service_pool.cc:225] Handling call
1023 02:04:50.763171 (+645881us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x7b5800002a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:04:51.087467 (+324296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2847104}
I20251023 02:04:50.995664 22589 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.088524 23296 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:50.819537 21856 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56864 (request call id 221) took 2573 ms. Trace:
I20251023 02:04:51.088989 23296 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.091915 23296 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.089825 21856 rpcz_store.cc:276] 1023 02:04:48.246203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.246419 (+ 216us) service_pool.cc:225] Handling call
1023 02:04:50.819461 (+2573042us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:04:49.478300 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:04:49.478836 (+ 536us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:04:49.478873 (+ 37us) write_op.cc:435] Acquired schema lock
1023 02:04:49.478883 (+ 10us) tablet.cc:660] Decoding operations
1023 02:04:49.641476 (+162593us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:04:49.645275 (+ 3799us) write_op.cc:641] Partition lock acquired for write op
1023 02:04:49.647158 (+ 1883us) tablet.cc:683] Acquiring locks for 6 operations
1023 02:04:49.801717 (+154559us) tablet.cc:699] Row locks acquired
1023 02:04:49.813608 (+ 11891us) write_op.cc:260] PREPARE: finished
1023 02:04:49.824335 (+ 10727us) write_op.cc:270] Start()
1023 02:04:49.835358 (+ 11023us) write_op.cc:276] Timestamp: P: 1761185089823349 usec, L: 0
1023 02:04:49.835386 (+ 28us) op_driver.cc:348] REPLICATION: starting
1023 02:04:49.890243 (+ 54857us) log.cc:844] Serialized 2436 byte log entry
1023 02:04:50.346829 (+456586us) op_driver.cc:464] REPLICATION: finished
1023 02:04:50.458374 (+111545us) write_op.cc:301] APPLY: starting
1023 02:04:50.458458 (+ 84us) tablet.cc:1365] starting BulkCheckPresence
1023 02:04:50.499699 (+ 41241us) tablet.cc:1368] finished BulkCheckPresence
1023 02:04:50.524308 (+ 24609us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:04:50.646148 (+121840us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:04:50.646426 (+ 278us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
1023 02:04:50.646494 (+ 68us) write_op.cc:312] APPLY: finished
1023 02:04:50.646798 (+ 304us) log.cc:844] Serialized 52 byte log entry
1023 02:04:50.647685 (+ 887us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:50.647821 (+ 136us) write_op.cc:454] Released schema lock
1023 02:04:50.647881 (+ 60us) write_op.cc:341] FINISH: Updating metrics
1023 02:04:50.648226 (+ 345us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:50.648252 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":108577,"apply.run_cpu_time_us":3653,"apply.run_wall_time_us":189847,"num_ops":6,"prepare.queue_time_us":173761,"prepare.run_cpu_time_us":5960,"prepare.run_wall_time_us":458076,"replication_time_us":495089,"thread_start_us":113770,"threads_started":2}]]}
W20251023 02:04:50.996863 23300 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.003s sys 0.007s
W20251023 02:04:51.093242 23300 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.660s user 0.004s sys 0.007s
I20251023 02:04:50.921989 23341 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:51.094779 23341 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.065979 23315 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.097247 23341 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:50.900350 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:50.993783 23092 raft_consensus.cc:740] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:51.103379 23092 consensus_queue.cc:260] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.104370 23092 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:50.970141 22028 rpcz_store.cc:276] 1023 02:04:49.032711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:49.325474 (+292763us) service_pool.cc:225] Handling call
1023 02:04:49.334795 (+ 9321us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:50.008267 (+673472us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:50.707753 (+699486us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b540020c1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:50.708017 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1536066432}
I20251023 02:04:51.000386 23202 raft_consensus.cc:3055] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:51.106029 23202 raft_consensus.cc:740] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:51.106771 23202 consensus_queue.cc:260] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.108139 23202 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:50.986883 23271 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:51.003931 23078 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.111856 22105 raft_consensus.cc:1240] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:51.114033 23078 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:51.004447 22077 leader_election.cc:336] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.593s (SENT)
I20251023 02:04:51.004863 23186 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:51.007189 23316 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:51.042292 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.048348 22034 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:50.798358 23026 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:50.859041 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.052726 23023 consensus_queue.cc:1048] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:50.989719 23248 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:50.869858 23331 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.092381 23296 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:50.997704 21955 raft_consensus.cc:606] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Received request to transfer leadership to TS 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:50.721266 22002 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.099166 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:51.089300 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 429414us
I20251023 02:04:51.159255 23315 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.160161 22589 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.082581 21858 catalog_manager.cc:5649] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 3, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 3 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251023 02:04:51.117769 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 113364us
I20251023 02:04:51.163859 21858 catalog_manager.cc:5649] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 2, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 2 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:04:51.122611 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:51.124073 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 62294.5us
I20251023 02:04:51.165634 22001 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.165784 23323 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.166574 22111 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
W20251023 02:04:51.130332 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 77092us
I20251023 02:04:51.131780 23297 consensus_queue.cc:1059] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:51.168740 23186 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:51.169207 23186 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.156678 23248 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.172331 23248 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.159369 22108 raft_consensus.cc:2393] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:51.147336 23026 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.167804 22589 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.182196 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.187453 22034 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
W20251023 02:04:51.190058 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 468918us
I20251023 02:04:51.195483 21925 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:51.197045 22002 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.189307 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.203642 23239 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:51.198866 23119 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:51.204173 23119 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:51.204536 23119 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:51.204739 23305 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:51.205137 23305 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:51.205502 23305 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.206382 22076 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:51.207579 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 146501us
I20251023 02:04:51.209051 22076 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.210665 22076 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.211922 22076 leader_election.cc:400] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:51.194628 23316 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:51.214267 23316 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.216231 23316 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:51.213748 23026 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.186226 23331 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.218605 22076 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.220993 21955 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6ead1daa70a847959916a302d8540676"
dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:40860
I20251023 02:04:51.221740 23078 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:51.222600 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.235649 22111 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:51.235894 22076 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:51.223713 22108 raft_consensus.cc:1240] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251023 02:04:51.224856 23315 raft_consensus.cc:993] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: : Instructing follower 1ab8b0b58147446d84d901b4ed2ff39e to start an election
I20251023 02:04:51.206022 22037 raft_consensus.cc:1240] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:51.231835 23082 raft_consensus.cc:3055] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:51.216424 22589 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.199122 23196 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:51.237926 21925 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:51.238095 22076 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.246416 22076 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.236936 23202 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.250218 22076 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.238708 23316 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.236586 23357 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.256351 21961 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20251023 02:04:51.241966 21955 raft_consensus.cc:606] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:51.260295 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 91577.7us
W20251023 02:04:51.263998 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:51.239894 23347 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:51.267345 22076 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.268543 22076 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:51.242363 23026 raft_consensus.cc:1081] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Signalling peer 1ab8b0b58147446d84d901b4ed2ff39e to start an election
I20251023 02:04:51.272239 22076 leader_election.cc:400] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:51.242611 23082 raft_consensus.cc:740] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:51.255769 23315 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:51.255549 23078 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251023 02:04:51.263530 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:51.275911 22076 leader_election.cc:403] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:04:51.256922 23296 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:51.254922 23319 raft_consensus.cc:2749] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.255064 23270 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.258286 23056 raft_consensus.cc:2749] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.283797 23087 raft_consensus.cc:3055] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:51.244597 23119 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.273187 23214 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:51.270411 23271 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:51.281330 23315 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:51.275134 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.272559 23216 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:51.291137 23087 raft_consensus.cc:740] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:51.304280 22622 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20251023 02:04:51.304597 22622 pending_rounds.cc:85] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting all ops after (but not including) 2
I20251023 02:04:51.304769 22622 pending_rounds.cc:107] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20251023 02:04:51.304971 22622 raft_consensus.cc:2917] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20251023 02:04:51.305282 22622 raft_consensus.cc:2889] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:04:51.301554 22619 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3"
dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
from {username='slave'} at 127.0.0.1:34806
I20251023 02:04:51.307435 22076 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.294221 23296 raft_consensus.cc:697] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:51.301117 23315 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.309756 23202 raft_consensus.cc:2749] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:51.314930 23087 consensus_queue.cc:260] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.346225 23248 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:51.346853 23087 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.346966 23248 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.316802 21955 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb"
dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:40860
I20251023 02:04:51.349972 21955 raft_consensus.cc:606] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.350241 23248 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:51.320626 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 123652us
I20251023 02:04:51.352032 22002 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:51.321741 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 123683us
I20251023 02:04:51.354746 23221 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:51.355373 21928 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.356954 21928 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.357800 23379 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:04:51.358366 21928 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.322350 23341 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.322981 23374 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20251023 02:04:51.325277 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 55354.1us
I20251023 02:04:51.326251 23331 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:51.325994 23373 consensus_queue.cc:1059] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:51.327729 23201 raft_consensus.cc:3055] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:51.327886 23347 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:51.327840 22076 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.328437 22619 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:51.394517 22619 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:04:51.394819 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 67100.9us
I20251023 02:04:51.333047 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:51.396557 22076 leader_election.cc:400] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:51.397738 23331 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.398072 23201 raft_consensus.cc:740] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:51.334693 23056 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:51.397317 22076 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.399180 23201 consensus_queue.cc:260] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.399634 23056 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:51.400031 23056 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:51.400352 23201 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.400396 23056 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:51.342536 23297 raft_consensus.cc:993] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14: : Instructing follower ca1362b9d9b34073bf35b6f6c9d1159a to start an election
I20251023 02:04:51.303475 23216 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.402400 23026 raft_consensus.cc:1081] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Signalling peer ca1362b9d9b34073bf35b6f6c9d1159a to start an election
I20251023 02:04:51.403666 23216 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:51.306954 21954 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:51.405004 23313 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.361088 23023 consensus_queue.cc:1059] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:51.355401 23221 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20251023 02:04:51.405730 23023 raft_consensus.cc:3055] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:51.406100 23023 raft_consensus.cc:740] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:51.362609 23304 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:51.406917 23304 raft_consensus.cc:697] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:51.406831 23023 consensus_queue.cc:260] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.408068 23023 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:51.407974 23304 consensus_queue.cc:237] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.363055 23248 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.360394 22000 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:51.410092 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 49771.4us
I20251023 02:04:51.361208 23373 raft_consensus.cc:3055] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:51.411743 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:51.412210 23373 raft_consensus.cc:740] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:51.412407 21954 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:51.412503 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.413314 23221 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:51.413743 23221 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:51.414153 23221 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:51.414708 21962 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75"
dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
mode: GRACEFUL
new_leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
from {username='slave'} at 127.0.0.1:40860
I20251023 02:04:51.415380 21962 raft_consensus.cc:606] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Received request to transfer leadership to TS 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:51.364822 21928 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.416363 22105 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6ead1daa70a847959916a302d8540676"
dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:39646
I20251023 02:04:51.416708 23026 raft_consensus.cc:2749] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:51.417043 22105 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:51.327723 23186 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.417500 22105 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:51.367002 23119 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.353467 22002 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.363658 23296 consensus_queue.cc:237] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.379357 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.420938 22589 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:51.421712 23119 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:51.422209 23119 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:51.422639 23119 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.423859 23378 raft_consensus.cc:2749] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:51.296676 23270 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.392068 23379 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.425050 23379 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.360357 22590 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:04:51.426784 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 66522.4us
I20251023 02:04:51.334537 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:51.412981 23373 consensus_queue.cc:260] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.428283 23373 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:51.429437 23092 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:51.429952 23092 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:51.430404 23092 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:51.433898 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 69199us
I20251023 02:04:51.434588 21928 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.434779 23323 consensus_queue.cc:1059] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:51.435693 21928 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.436777 21928 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.437681 23186 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:51.438226 23379 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:51.438562 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 85214.3us
I20251023 02:04:51.424731 23305 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.439342 22002 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.440165 23379 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:51.440413 23305 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.440532 23379 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:51.441044 23379 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.443110 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.444039 21955 raft_consensus.cc:2410] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:04:51.446085 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.447836 22037 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251023 02:04:51.449290 21961 raft_consensus.cc:1275] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:51.386171 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:51.256335 23336 consensus_queue.cc:1059] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20251023 02:04:51.460016 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.455103 23270 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:51.449422 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.474771 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.475831 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.476351 22030 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:51.477041 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.477739 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:51.454087 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.458977 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.479215 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.459059 21928 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.459263 23313 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:51.455677 23026 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:51.440414 22002 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.505738 22036 raft_consensus.cc:2393] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
W20251023 02:04:51.506140 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 65799.3us
I20251023 02:04:51.455456 23248 raft_consensus.cc:2749] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:51.449396 23092 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.519961 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.520432 23092 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.520900 23315 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.521358 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.521657 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.522701 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:51.522929 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.524420 22104 raft_consensus.cc:1240] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:51.452242 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.459307 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:51.545285 21961 raft_consensus.cc:2393] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:51.473450 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:51.489665 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.546111 22111 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:51.490846 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.493261 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.503293 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.510267 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:51.510625 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:51.511143 23119 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.507395 23102 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:51.511508 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:51.569058 21958 raft_consensus.cc:2393] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:51.512542 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.513263 23166 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:51.515553 23056 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.517534 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.517853 21959 raft_consensus.cc:2410] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:04:51.518271 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.461265 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:51.549773 22030 raft_consensus.cc:2393] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:51.528347 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:51.550467 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:51.504739 23297 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:51.453889 23379 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:51.578101 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:51.575634 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.578796 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.581306 22077 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.583214 22036 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:51.565856 22037 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:51.615759 22030 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20251023 02:04:51.559239 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.525046 23270 consensus_queue.cc:1048] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20251023 02:04:51.567983 23300 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.556035 22618 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:51.624615 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.572538 23166 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:51.646281 21958 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:51.556594 23352 consensus_queue.cc:1059] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20251023 02:04:51.647156 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.647816 21958 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:51.648708 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.632349 21955 raft_consensus.cc:2368] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4.
I20251023 02:04:51.572877 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:51.581792 23297 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:51.617230 22615 raft_consensus.cc:2368] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4.
I20251023 02:04:51.639240 22111 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:51.640215 22077 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:51.647056 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.548657 23309 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20251023 02:04:51.651448 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.634548 22590 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:51.655928 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:51.652526 22031 raft_consensus.cc:2393] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
W20251023 02:04:51.649147 23364 log.cc:927] Time spent T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20251023 02:04:51.657305 23297 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.659904 23352 raft_consensus.cc:3055] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:51.678028 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.660195 23119 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.572431 23202 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20251023 02:04:51.665525 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:51.712548 22075 leader_election.cc:400] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255) with higher term. Message: Invalid argument: T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4.
I20251023 02:04:51.712767 22111 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:51.713048 22075 leader_election.cc:403] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:04:51.715458 22111 raft_consensus.cc:1275] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:51.716904 22111 raft_consensus.cc:1275] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:51.624929 22103 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:51.579361 23102 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:51.672999 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:51.669381 22033 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:51.601292 23201 consensus_queue.cc:1048] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.100s
I20251023 02:04:51.646075 23305 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.723341 23305 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.669505 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.726997 21958 raft_consensus.cc:2393] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 2: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:51.639202 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:51.649840 23166 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.727260 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.694084 23221 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.678272 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:51.696868 23315 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.697841 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.697947 22619 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.700053 21954 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:51.702203 22034 raft_consensus.cc:2410] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:04:51.702219 23313 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:51.745339 23313 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.702695 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.702836 23379 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.649425 22102 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:51.703171 23311 consensus_queue.cc:1059] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:04:51.649119 23300 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.704530 23056 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:51.674969 23186 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:51.678781 23092 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:51.710357 22105 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.711278 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:51.722026 23270 consensus_queue.cc:1059] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:04:51.728233 23216 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.733206 23102 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:51.734318 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:51.678344 23352 raft_consensus.cc:740] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:51.736368 22617 raft_consensus.cc:2393] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:51.690692 23242 consensus_queue.cc:1059] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20251023 02:04:51.725402 23305 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.780761 23092 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:51.729615 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.816797 21958 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:51.741344 23026 raft_consensus.cc:2749] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.743268 22028 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:51.743705 22619 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.744853 21954 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
W20251023 02:04:51.795701 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 289544us
I20251023 02:04:51.776074 23315 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.746481 22587 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.828164 21958 raft_consensus.cc:1275] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:51.788452 23186 raft_consensus.cc:697] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:51.831967 22075 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.818111 23092 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:51.862107 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 119937us
I20251023 02:04:51.864143 22587 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:51.865502 23323 raft_consensus.cc:2804] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:51.866110 23323 raft_consensus.cc:697] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:51.867585 21959 raft_consensus.cc:1275] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:51.867287 23323 consensus_queue.cc:237] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.868495 23186 consensus_queue.cc:237] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:51.873297 23392 log.cc:927] Time spent T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.162s user 0.002s sys 0.000s
I20251023 02:04:51.873840 23026 consensus_queue.cc:1059] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:51.876585 23379 raft_consensus.cc:3055] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:51.877135 23379 raft_consensus.cc:740] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:51.877879 23379 consensus_queue.cc:260] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.880352 22105 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:51.876648 23300 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.888839 22615 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
W20251023 02:04:51.889884 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 381563us
I20251023 02:04:51.854863 23297 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.906874 22076 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.904982 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1139) took 1186 ms. Trace:
I20251023 02:04:51.884903 23319 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.907987 22109 rpcz_store.cc:276] 1023 02:04:50.718618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.005870 (+287252us) service_pool.cc:225] Handling call
1023 02:04:51.006225 (+ 355us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:51.240616 (+234391us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:51.240644 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:51.243752 (+ 3108us) log.cc:844] Serialized 26 byte log entry
1023 02:04:51.650672 (+406920us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:51.650714 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:51.898384 (+247670us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54000fc154. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:51.898443 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:51.898496 (+ 53us) raft_consensus.cc:1685] finished
1023 02:04:51.898517 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:51.904905 (+ 6388us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":391,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":15710,"spinlock_wait_cycles":538409728,"thread_start_us":376077,"threads_started":1,"wal-append.queue_time_us":486662}
I20251023 02:04:51.913089 22076 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.896692 23395 raft_consensus.cc:3055] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
W20251023 02:04:51.904630 23366 log.cc:927] Time spent T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20251023 02:04:51.917176 22076 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:51.917721 22589 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:51.928488 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 291005us
I20251023 02:04:51.849030 23315 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:51.923915 21961 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:51.911926 23331 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:51.943670 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:51.904107 23304 consensus_queue.cc:1059] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20251023 02:04:51.920302 23270 raft_consensus.cc:2749] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:51.920778 23221 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:51.923230 23253 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.958712 22076 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.919701 23395 raft_consensus.cc:740] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:51.965000 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:51.920958 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:51.931517 23005 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:51.972663 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:51.974442 22076 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:51.975916 22076 leader_election.cc:400] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4.
I20251023 02:04:51.976254 22076 leader_election.cc:403] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:04:51.973484 21959 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:51.940943 23397 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:51.978842 21954 raft_consensus.cc:1275] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:51.979024 22002 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.980243 21954 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:51.918282 23271 consensus_queue.cc:1048] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20251023 02:04:51.903579 23352 consensus_queue.cc:260] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:51.987287 21962 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:51.983035 22077 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:51.978886 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:51.944336 23242 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:51.993484 22002 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:52.004635 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:04:51.994051 22000 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:52.007706 22002 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:52.027057 22002 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:51.983457 23270 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.973464 23331 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:51.990514 23315 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:51.993139 23214 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:51.998435 23202 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.038220 22035 raft_consensus.cc:1275] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:52.040033 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:51.998147 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 65912.6us
I20251023 02:04:51.998306 23375 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:52.041335 22000 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:52.048935 22035 raft_consensus.cc:2393] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:51.982367 23253 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.028604 23221 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:52.032992 22002 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:51.997355 23005 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.029364 23376 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:52.055245 22035 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20251023 02:04:52.049336 23119 raft_consensus.cc:3055] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:52.035228 23404 raft_consensus.cc:3055] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:52.037518 23023 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:52.035152 23214 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:52.035297 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:52.030000 23405 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.043478 23270 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:52.042407 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:52.050981 23375 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:04:52.105619 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 50115.4us
W20251023 02:04:52.097508 23394 log.cc:927] Time spent T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20251023 02:04:52.099587 23407 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:52.107788 23404 raft_consensus.cc:740] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:04:52.110208 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 237379us
I20251023 02:04:52.107471 23315 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:52.164917 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 132009us
I20251023 02:04:52.165990 22002 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:52.104894 23119 raft_consensus.cc:740] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:52.056723 23395 consensus_queue.cc:260] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.115826 23214 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:52.178718 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 202971us
I20251023 02:04:52.181730 22076 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:52.182869 23270 raft_consensus.cc:2749] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251023 02:04:52.187884 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 59459.1us
I20251023 02:04:52.128842 23375 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.130582 23407 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:52.174674 23297 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:52.193452 23401 log.cc:927] Time spent T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.137s user 0.000s sys 0.000s
W20251023 02:04:52.214184 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:52.191520 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:52.193820 23406 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:52.229924 23248 raft_consensus.cc:993] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14: : Instructing follower ca1362b9d9b34073bf35b6f6c9d1159a to start an election
I20251023 02:04:52.203312 23253 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:52.203470 23409 log.cc:927] Time spent T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20251023 02:04:52.252751 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:52.199275 23376 raft_consensus.cc:2749] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:52.204528 23239 consensus_queue.cc:1048] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:52.253408 22103 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:52.207533 23087 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:52.254383 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:52.254966 22106 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.126952 23082 consensus_queue.cc:260] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.254793 23087 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.208889 23221 consensus_queue.cc:1048] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251023 02:04:52.193637 23419 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20251023 02:04:52.121519 23296 consensus_queue.cc:1059] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
W20251023 02:04:52.226048 23403 log.cc:927] Time spent T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
W20251023 02:04:52.202452 23410 log.cc:927] Time spent T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20251023 02:04:52.242100 23005 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.211084 23407 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.126437 23405 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.228044 23395 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:52.262060 23379 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.262343 23087 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:52.263346 23389 log.cc:927] Time spent T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.565s user 0.001s sys 0.002s
I20251023 02:04:52.263953 23092 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.194648 23023 consensus_queue.cc:1048] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20251023 02:04:52.245253 23352 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:52.251637 23426 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.291532 22621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 159) took 1015 ms. Trace:
I20251023 02:04:52.254890 23304 raft_consensus.cc:1081] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Signalling peer ca1362b9d9b34073bf35b6f6c9d1159a to start an election
I20251023 02:04:52.226464 23377 consensus_queue.cc:1048] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20251023 02:04:52.188434 23315 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:52.313510 23388 log.cc:927] Time spent T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.621s user 0.001s sys 0.002s
I20251023 02:04:52.313568 23315 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:52.314549 23056 raft_consensus.cc:3055] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Stepping down as leader of term 1
W20251023 02:04:52.314860 23313 consensus_meta.cc:287] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.550s user 0.010s sys 0.000s
I20251023 02:04:52.314980 23056 raft_consensus.cc:740] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:52.315286 23313 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.315814 23056 consensus_queue.cc:260] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.316773 23056 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:52.317189 23313 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.249171 23404 consensus_queue.cc:260] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.298197 23406 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:04:52.299221 23393 log.cc:927] Time spent T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.602s user 0.000s sys 0.002s
I20251023 02:04:52.302673 23375 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.321561 23375 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:52.304332 23166 consensus_meta.cc:287] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.562s user 0.011s sys 0.000s
I20251023 02:04:52.322032 23082 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.322353 23166 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.324290 23166 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.305917 23405 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.325141 23426 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.325904 23404 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:52.327261 23426 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.306836 23092 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.308037 23119 consensus_queue.cc:260] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.328542 23119 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.292868 22621 rpcz_store.cc:276] 1023 02:04:51.292286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.295123 (+ 2837us) service_pool.cc:225] Handling call
1023 02:04:51.295549 (+ 426us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:51.297607 (+ 2058us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:51.297654 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:51.298060 (+ 406us) log.cc:844] Serialized 26 byte log entry
1023 02:04:51.627512 (+329452us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:51.627569 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:51.627659 (+ 90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:52.262803 (+635144us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:04:52.290628 (+ 27825us) raft_consensus.cc:1685] finished
1023 02:04:52.290652 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:52.291468 (+ 816us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4692,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":4517,"spinlock_wait_cycles":84160128,"thread_start_us":325289,"threads_started":1,"wal-append.queue_time_us":396723}
I20251023 02:04:52.300997 23214 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.339239 23214 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.291342 23216 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255) with higher term. Message: Invalid argument: T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4.
I20251023 02:04:52.344731 23407 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.345878 23202 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4.
I20251023 02:04:52.355969 23407 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.208312 22002 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:52.268999 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:52.394906 22108 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:52.225380 22028 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:04:52.391311 21956 raft_consensus.cc:2410] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251023 02:04:52.391762 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:52.264273 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:52.266621 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:52.199188 22032 raft_consensus.cc:2393] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:52.442423 23404 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
W20251023 02:04:52.394909 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 186569us
I20251023 02:04:52.484778 22035 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:52.492673 23407 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:52.493327 23407 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:52.492604 23404 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.493731 23407 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.495896 23404 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:52.499579 23377 consensus_queue.cc:1048] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:52.499864 23406 consensus_queue.cc:1059] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:52.501036 23406 consensus_queue.cc:1243] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251023 02:04:52.501969 23196 raft_consensus.cc:3055] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:52.502514 23196 raft_consensus.cc:740] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:52.503926 23196 consensus_queue.cc:260] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:52.571705 21912 thread.cc:641] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.016s sys 0.048s
W20251023 02:04:52.572252 21912 thread.cc:608] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.582s user 0.016s sys 0.049s
I20251023 02:04:52.574923 23092 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:04:52.576103 22034 raft_consensus.cc:1240] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251023 02:04:52.575027 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:52.583859 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 883) took 1123 ms. Trace:
I20251023 02:04:52.584379 21962 rpcz_store.cc:276] 1023 02:04:51.460147 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.461044 (+ 897us) service_pool.cc:225] Handling call
1023 02:04:52.583320 (+1122276us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b54002a08d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:52.583584 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":629353344}
W20251023 02:04:52.585727 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:52.586992 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 882) took 1127 ms. Trace:
I20251023 02:04:52.587558 21956 rpcz_store.cc:276] 1023 02:04:51.459847 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.478359 (+ 18512us) service_pool.cc:225] Handling call
1023 02:04:52.586656 (+1108297us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b5400301e54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:52.586909 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1666004352}
I20251023 02:04:52.588055 23352 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:52.589123 23216 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.590075 23216 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.590538 23214 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:52.591421 23319 raft_consensus.cc:1064] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:52.591504 23214 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.592327 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 886) took 1131 ms. Trace:
I20251023 02:04:52.592871 23216 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.588910 23352 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.592764 21957 rpcz_store.cc:276] 1023 02:04:51.460943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.470494 (+ 9551us) service_pool.cc:225] Handling call
1023 02:04:51.512340 (+ 41846us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:51.548131 (+ 35791us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:51.548162 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:51.557427 (+ 9265us) log.cc:844] Serialized 26 byte log entry
1023 02:04:51.918195 (+360768us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:52.101297 (+183102us) log.cc:844] Serialized 14 byte log entry
1023 02:04:52.591753 (+490456us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:52.591883 (+ 130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:52.591960 (+ 77us) raft_consensus.cc:1685] finished
1023 02:04:52.591973 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:52.592135 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66749,"prepare.queue_time_us":352323,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":170308,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":36840,"thread_start_us":892410,"threads_started":3,"wal-append.queue_time_us":416281}
W20251023 02:04:52.592118 23319 raft_consensus.cc:1068] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: Unable to promote non-voter 1ab8b0b58147446d84d901b4ed2ff39e: Illegal state: Leader has not yet committed an operation in its own term
I20251023 02:04:52.597020 23352 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.597290 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1128) took 1021 ms. Trace:
I20251023 02:04:52.600091 21958 rpcz_store.cc:276] 1023 02:04:51.575346 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.844880 (+269534us) service_pool.cc:225] Handling call
1023 02:04:51.845178 (+ 298us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:51.845370 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:04:52.132587 (+287217us) log.cc:844] Serialized 14 byte log entry
1023 02:04:52.596414 (+463827us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:52.596556 (+ 142us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:52.596724 (+ 168us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:52.596853 (+ 129us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:52.597073 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":424743,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":135,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":259,"thread_start_us":711237,"threads_started":2}
I20251023 02:04:52.601322 22621 raft_consensus.cc:1240] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:52.609524 23352 consensus_queue.cc:1059] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.009s
I20251023 02:04:52.622052 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 890) took 1116 ms. Trace:
W20251023 02:04:52.626319 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:52.627753 22111 raft_consensus.cc:2410] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:04:52.625635 21961 rpcz_store.cc:276] 1023 02:04:51.494250 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.738679 (+244429us) service_pool.cc:225] Handling call
1023 02:04:51.766780 (+ 28101us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:51.932846 (+166066us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:52.528643 (+595797us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54000c5154. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:52.610636 (+ 81993us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":352603008}
W20251023 02:04:52.626973 22033 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s user 0.008s sys 0.033s
I20251023 02:04:52.632484 23196 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:52.632139 23397 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.032s sys 0.033s
W20251023 02:04:52.634397 22622 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.021s sys 0.019s
W20251023 02:04:52.638872 22107 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.038s sys 0.045s
W20251023 02:04:52.642997 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:52.645300 22622 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.693s user 0.022s sys 0.019s
W20251023 02:04:52.647284 22107 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.039s sys 0.045s
W20251023 02:04:52.653230 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 450633us
W20251023 02:04:52.654049 23397 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.032s sys 0.033s
W20251023 02:04:52.663944 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 406582us
I20251023 02:04:52.673995 23430 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.674880 23430 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:52.671372 21925 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.008s sys 0.010s
W20251023 02:04:52.682094 21925 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.565s user 0.009s sys 0.010s
I20251023 02:04:52.693847 23442 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.662997 23407 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.679606 23397 raft_consensus.cc:697] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:52.710448 23056 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.711148 23375 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:52.716427 23439 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.716199 21928 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:52.694808 23442 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.718793 23430 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:52.704129 21953 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.009s sys 0.019s
I20251023 02:04:52.727931 23297 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.731324 23297 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.728252 23319 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:52.732759 23319 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.700698 23352 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:52.733863 23119 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:52.734673 23119 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.735059 23319 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:52.734236 23352 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.736073 23379 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:52.736939 23119 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.736745 23379 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.738708 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.739912 22033 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:52.740638 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:52.742123 22590 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:52.742642 23375 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.743144 23119 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:52.743486 23119 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:52.743858 23119 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:52.744904 23375 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.745136 23352 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.747728 22622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 205) took 1258 ms. Trace:
I20251023 02:04:52.738991 23379 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.746031 23056 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:52.763331 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 70339us
I20251023 02:04:52.765064 23440 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:52.765767 23404 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.766384 23347 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:52.754726 21953 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.762s user 0.009s sys 0.020s
W20251023 02:04:52.773003 22106 consensus_meta.cc:287] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.517s user 0.009s sys 0.005s
I20251023 02:04:52.755491 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1136) took 1087 ms. Trace:
I20251023 02:04:52.768761 23404 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.773468 22109 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:04:52.774914 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 89) took 1259 ms. Trace:
I20251023 02:04:52.782960 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:52.775933 23214 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:52.781774 22106 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:04:52.748499 22622 rpcz_store.cc:276] 1023 02:04:51.488955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.490951 (+ 1996us) service_pool.cc:225] Handling call
1023 02:04:51.491842 (+ 891us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:51.527341 (+ 35499us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:51.527483 (+ 142us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:51.563493 (+ 36010us) log.cc:844] Serialized 26 byte log entry
1023 02:04:51.910221 (+346728us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:04:52.696380 (+786159us) log.cc:844] Serialized 14 byte log entry
1023 02:04:52.747017 (+ 50637us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:52.747155 (+ 138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:52.747275 (+ 120us) raft_consensus.cc:1685] finished
1023 02:04:52.747286 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:52.747557 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58058,"prepare.queue_time_us":53087,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":68015,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":6710,"thread_start_us":955414,"threads_started":2,"wal-append.queue_time_us":412566}
I20251023 02:04:52.778267 22107 rpcz_store.cc:276] 1023 02:04:51.510886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.523697 (+ 12811us) service_pool.cc:225] Handling call
1023 02:04:51.524015 (+ 318us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:51.524241 (+ 226us) raft_consensus.cc:1534] Early marking committed up to index 5
1023 02:04:51.782330 (+258089us) log.cc:844] Serialized 14 byte log entry
1023 02:04:52.684749 (+902419us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:52.697509 (+ 12760us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:04:52.723637 (+ 26128us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:52.753071 (+ 29434us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:52.770437 (+ 17366us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37819,"prepare.queue_time_us":390371,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":1890,"raft.queue_time_us":23907,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"thread_start_us":1122492,"threads_started":3,"wal-append.queue_time_us":344674}
I20251023 02:04:52.783252 23439 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.799201 22000 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:52.799149 23297 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.800141 23214 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:52.814250 23407 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.822964 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:52.826943 23442 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.835765 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:52.838122 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:52.829421 23375 raft_consensus.cc:2749] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:52.841701 23404 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:52.836715 23352 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:52.836887 23056 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:52.843241 22103 consensus_meta.cc:287] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.549s user 0.016s sys 0.000s
I20251023 02:04:52.849378 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:52.776260 21955 rpcz_store.cc:276] 1023 02:04:51.584388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.867502 (+283114us) service_pool.cc:225] Handling call
1023 02:04:51.867808 (+ 306us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:51.867985 (+ 177us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:04:52.190766 (+322781us) log.cc:844] Serialized 14 byte log entry
1023 02:04:52.619137 (+428371us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:52.627272 (+ 8135us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:52.631509 (+ 4237us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:52.641241 (+ 9732us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:52.672140 (+ 30899us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24120,"prepare.queue_time_us":402324,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":1958,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":12988,"thread_start_us":438731,"threads_started":2,"wal-append.queue_time_us":247836}
I20251023 02:04:52.858141 21961 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.842185 23214 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.861016 23433 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.860934 23439 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.849812 23202 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:52.849488 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.832789 23397 consensus_queue.cc:237] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:52.867308 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 79034.3us
I20251023 02:04:52.859122 23444 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:52.870388 23202 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:52.873850 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:52.862972 23404 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:52.867097 22103 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
W20251023 02:04:52.881744 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 49957.3us
I20251023 02:04:52.880255 22000 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:52.885913 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:52.876602 23404 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.875703 23352 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.893055 23297 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.893186 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:52.893905 21928 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:52.893723 23297 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.875572 23444 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.899067 23437 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:52.899230 23297 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.882943 22031 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:52.898319 21962 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:52.902340 21955 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:52.903189 22590 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:52.903765 23202 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.903515 23119 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.905784 23214 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.883117 23433 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.906164 22587 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:52.908990 23433 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:52.895756 23056 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:52.909293 23216 raft_consensus.cc:3055] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:52.911240 22033 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:04:52.907704 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:52.911789 23444 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:52.935098 22031 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:52.936959 22033 raft_consensus.cc:606] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:52.938386 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.939096 22033 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:52.936980 23319 consensus_queue.cc:1048] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20251023 02:04:52.912075 23442 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:52.941045 23442 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:52.941461 23442 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.941891 23319 consensus_queue.cc:1059] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:52.943576 23319 raft_consensus.cc:3055] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:52.943950 23319 raft_consensus.cc:740] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:52.945739 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20251023 02:04:52.945729 23319 consensus_queue.cc:260] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.918540 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:52.948813 23319 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.949215 22102 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:52.950284 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:52.950775 22102 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.913422 23404 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:52.919080 22108 raft_consensus.cc:1240] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:52.918831 23352 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.919955 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:52.954591 22111 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:52.954783 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:52.955469 22108 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:52.955528 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:52.920526 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:52.922538 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:52.923651 22034 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:52.956470 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:52.956708 22106 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:52.919482 23092 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:52.957581 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.957947 23092 raft_consensus.cc:697] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:52.958091 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:52.958808 22106 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.958974 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.959309 23092 consensus_queue.cc:237] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.924089 22029 raft_consensus.cc:1240] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:52.924419 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.925581 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:52.926317 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.891551 23196 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:52.927716 22036 raft_consensus.cc:1275] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:52.931592 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:52.961187 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.912004 21961 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:04:52.961294 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:52.940069 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.926481 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.961890 22035 raft_consensus.cc:2393] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:52.962172 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.962955 22037 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251023 02:04:52.932914 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:52.907857 23119 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.940022 23056 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.973017 22077 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:52.975520 23214 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:52.975610 23428 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:52.976286 23214 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:52.976564 22077 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:52.977100 23056 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.977756 22033 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:52.977881 23426 consensus_queue.cc:1059] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:52.978340 23214 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:52.978663 22030 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:52.978950 23214 raft_consensus.cc:3055] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:52.979440 23270 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:52.979693 23214 raft_consensus.cc:740] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:52.979882 23270 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:52.979931 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:52.980620 23270 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.980929 21954 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:52.981338 22111 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:52.981246 23214 consensus_queue.cc:260] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:52.978729 23426 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:52.974236 22029 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:52.988160 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:52.990032 22034 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:52.991421 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:52.934020 23216 raft_consensus.cc:740] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:52.992556 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:52.975024 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:52.993541 23216 consensus_queue.cc:260] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:52.994225 22035 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:52.994715 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:52.994922 23216 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.995427 23214 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:52.995663 21960 raft_consensus.cc:2410] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20251023 02:04:52.983551 22106 raft_consensus.cc:2410] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251023 02:04:52.984369 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 90618.8us
W20251023 02:04:52.998871 23448 log.cc:927] Time spent T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.138s user 0.003s sys 0.000s
I20251023 02:04:52.998890 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:52.986466 21957 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 1.
I20251023 02:04:52.999498 21928 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.012934 22108 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:53.016510 22104 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:04:53.026611 22077 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:53.029081 23450 log.cc:927] Time spent T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.160s user 0.001s sys 0.000s
I20251023 02:04:53.030257 23352 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:53.031644 23352 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:53.031965 23352 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:53.032281 23352 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.033044 23456 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:53.033610 23456 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.034934 22107 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251023 02:04:53.035764 23456 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:53.036095 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:53.016511 23433 raft_consensus.cc:2749] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:53.037513 21960 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20251023 02:04:53.037918 21960 pending_rounds.cc:85] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 2
I20251023 02:04:53.038149 21960 pending_rounds.cc:107] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20251023 02:04:53.038452 21960 raft_consensus.cc:2917] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20251023 02:04:53.038916 21960 raft_consensus.cc:2889] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:04:53.020622 23119 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:53.042645 22105 raft_consensus.cc:2410] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:04:53.042383 23119 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.042673 22077 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.045734 22000 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.048650 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.050856 22077 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.053781 22077 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.054373 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.055015 22077 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:53.055420 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:53.025553 23426 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:53.057477 23271 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:53.057544 22034 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:04:53.058277 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:53.043731 23270 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.058446 21955 raft_consensus.cc:1240] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251023 02:04:53.059060 23375 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:53.059785 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:53.059890 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 60581.5us
I20251023 02:04:53.047895 23347 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251023 02:04:53.061246 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 53531.3us
I20251023 02:04:53.066053 23379 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:53.066303 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:53.068634 22590 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:53.024966 23442 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.046674 23056 consensus_queue.cc:1059] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
W20251023 02:04:53.049949 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 62274.3us
I20251023 02:04:53.071163 21957 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:53.053049 23202 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:53.074838 23455 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:53.068123 23271 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:53.078034 23271 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:53.060575 23319 raft_consensus.cc:2804] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:53.073998 22619 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:53.074368 21953 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20251023 02:04:53.084488 23426 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.086158 21953 pending_rounds.cc:85] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 3
I20251023 02:04:53.086510 21953 pending_rounds.cc:107] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20251023 02:04:53.086789 21953 raft_consensus.cc:2889] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:04:53.087751 23426 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:53.088001 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:53.083763 23119 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.076124 23456 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:53.089021 23456 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:53.089382 23456 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.090711 22000 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.091567 23439 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:53.091682 22000 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.092031 23439 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:53.092430 23439 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.092823 22000 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.093642 23374 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:53.059630 23214 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:53.075251 21961 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.058445 22102 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:53.085377 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.083106 23319 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:53.090894 23271 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.103143 23319 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:53.103188 23407 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:53.103713 23407 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:53.104049 23407 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.104970 23271 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:53.106128 23456 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.096246 21959 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:53.066349 23352 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.071661 23379 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:53.097831 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.108279 23456 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.093998 23374 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:53.113778 23374 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.078034 23455 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.102717 21960 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:53.076822 23202 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:53.116384 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.116482 21958 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.116515 23202 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.118043 23455 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.118310 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.118754 22587 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.119159 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.119705 21960 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:53.119736 22587 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:53.120626 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:53.108525 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251023 02:04:53.124619 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 43826.4us
W20251023 02:04:53.128116 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:53.129722 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.096925 23214 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:53.142688 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:53.143301 21960 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.143965 22000 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.115677 23379 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.082930 23375 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.146152 21961 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:53.124121 21953 raft_consensus.cc:1275] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:53.094070 23196 consensus_queue.cc:1059] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.045s
I20251023 02:04:53.152279 23270 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:53.156282 23319 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.157646 23397 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:53.157872 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.158912 21954 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:53.162132 23202 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.097810 23426 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:53.165977 23426 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.166281 23442 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:53.167888 23426 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:53.168713 23375 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.170156 23202 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:53.171072 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.175225 23196 raft_consensus.cc:3055] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:53.175601 23196 raft_consensus.cc:740] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:53.176276 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.176553 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.176455 23196 consensus_queue.cc:260] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.177918 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.179077 21959 raft_consensus.cc:1275] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:53.171151 22031 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:53.172505 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.180312 21957 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
W20251023 02:04:53.172829 23453 log.cc:927] Time spent T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20251023 02:04:53.147951 21928 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:53.158944 23457 log.cc:927] Time spent T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20251023 02:04:53.181973 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.182929 21928 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.183727 23404 raft_consensus.cc:2749] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:53.184442 23404 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:53.184930 23404 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:53.185333 23404 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.186669 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.186885 23352 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:53.187418 22035 raft_consensus.cc:2393] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:53.133749 23455 raft_consensus.cc:2804] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:53.184260 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.141726 23214 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.192592 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.193689 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.193718 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.199363 22035 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.200178 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.200987 22034 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.204551 21958 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:53.186693 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.207511 23378 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:53.191442 23456 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251023 02:04:53.208112 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:53.208751 21957 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:53.211103 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.203583 23196 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.204453 23455 raft_consensus.cc:697] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:53.215838 23319 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:53.234901 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.212970 23378 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:53.240307 23374 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.143350 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251023 02:04:53.250257 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 106355us
I20251023 02:04:53.226706 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.254971 21960 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
W20251023 02:04:53.264119 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 72021.7us
I20251023 02:04:53.261338 22031 raft_consensus.cc:2410] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251023 02:04:53.250156 23378 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.264688 22590 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:04:53.246609 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 125612us
I20251023 02:04:53.269398 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.270895 23407 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.277731 22029 raft_consensus.cc:1240] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:53.278739 23407 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.279047 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:53.280400 22107 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb"
dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:39646
I20251023 02:04:53.281036 22107 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:53.281394 22107 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:53.281847 21858 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33304 (request call id 91) took 5142 ms. Trace:
I20251023 02:04:53.281901 22103 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:53.283210 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:53.272048 22104 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56894 (request call id 90) took 1690 ms (client timeout 1553 ms). Trace:
I20251023 02:04:53.272576 21954 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:53.284407 23005 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:53.285234 23442 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:53.285495 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.286480 23442 raft_consensus.cc:697] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:53.286631 23455 consensus_queue.cc:237] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.287461 23442 consensus_queue.cc:237] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.288575 22028 raft_consensus.cc:1240] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:53.288918 23379 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:53.284129 22104 rpcz_store.cc:269] 1023 02:04:51.582329 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.648285 (+ 65956us) service_pool.cc:225] Handling call
1023 02:04:53.270573 (+1622288us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b540025b7d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:53.271936 (+ 1363us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1568290176}
I20251023 02:04:53.284965 23005 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.289973 22619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 169) took 1306 ms. Trace:
I20251023 02:04:53.276448 23214 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.267248 21961 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:53.290095 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.293797 22000 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.296440 22109 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:53.299165 22035 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:04:53.266224 23444 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:53.300120 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:53.301911 22076 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:53.302563 23374 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.306929 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:53.311902 22621 raft_consensus.cc:2393] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:53.312062 22034 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:04:53.316960 23444 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.312073 22002 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.317761 23166 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:53.318310 23166 raft_consensus.cc:697] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:53.319180 23444 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.319268 23166 consensus_queue.cc:237] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.318418 23385 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251023 02:04:53.323205 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 59051.4us
I20251023 02:04:53.327086 22035 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:53.330861 22111 raft_consensus.cc:1240] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:53.328945 22107 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.338963 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:53.334476 22590 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:53.323902 22002 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.341094 22107 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:53.340981 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.271687 23439 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.343071 23319 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:53.344401 23271 raft_consensus.cc:3055] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:53.345801 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.282348 21858 rpcz_store.cc:276] 1023 02:04:48.104286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:48.104482 (+ 196us) service_pool.cc:225] Handling call
1023 02:04:53.245569 (+5141087us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:04:51.690682 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:04:51.693423 (+ 2741us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:04:51.693472 (+ 49us) write_op.cc:435] Acquired schema lock
1023 02:04:51.693496 (+ 24us) tablet.cc:660] Decoding operations
1023 02:04:51.694230 (+ 734us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:04:51.694311 (+ 81us) write_op.cc:641] Partition lock acquired for write op
1023 02:04:51.694350 (+ 39us) tablet.cc:683] Acquiring locks for 3 operations
1023 02:04:51.694774 (+ 424us) tablet.cc:699] Row locks acquired
1023 02:04:51.694798 (+ 24us) write_op.cc:260] PREPARE: finished
1023 02:04:51.694979 (+ 181us) write_op.cc:270] Start()
1023 02:04:51.695116 (+ 137us) write_op.cc:276] Timestamp: P: 1761185091694923 usec, L: 0
1023 02:04:51.695143 (+ 27us) op_driver.cc:348] REPLICATION: starting
1023 02:04:51.695906 (+ 763us) log.cc:844] Serialized 1283 byte log entry
1023 02:04:51.880640 (+184734us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5c9cdfdc 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c5f62f2bb
1023 02:04:52.686820 (+806180us) op_driver.cc:464] REPLICATION: finished
1023 02:04:52.843820 (+157000us) write_op.cc:301] APPLY: starting
1023 02:04:52.851648 (+ 7828us) tablet.cc:1365] starting BulkCheckPresence
1023 02:04:52.871723 (+ 20075us) tablet.cc:1368] finished BulkCheckPresence
1023 02:04:52.874168 (+ 2445us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:04:52.936598 (+ 62430us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:04:52.936879 (+ 281us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1023 02:04:52.936958 (+ 79us) write_op.cc:312] APPLY: finished
1023 02:04:52.937336 (+ 378us) log.cc:844] Serialized 34 byte log entry
1023 02:04:52.938764 (+ 1428us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:52.938983 (+ 219us) write_op.cc:454] Released schema lock
1023 02:04:52.939076 (+ 93us) write_op.cc:341] FINISH: Updating metrics
1023 02:04:52.941288 (+ 2212us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:52.941319 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":121184,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":141402,"apply.run_cpu_time_us":3626,"apply.run_wall_time_us":106293,"num_ops":3,"prepare.queue_time_us":473762,"prepare.run_cpu_time_us":37406,"prepare.run_wall_time_us":384270,"raft.queue_time_us":440917,"raft.run_cpu_time_us":12639,"raft.run_wall_time_us":70756,"replication_time_us":991390,"spinlock_wait_cycles":98767104,"thread_start_us":870337,"threads_started":4,"wal-append.queue_time_us":484512}]]}
I20251023 02:04:53.347290 23433 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:53.348121 23458 log.cc:927] Time spent T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20251023 02:04:53.303328 22619 rpcz_store.cc:276] 1023 02:04:51.983597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.993382 (+ 9785us) service_pool.cc:225] Handling call
1023 02:04:52.001145 (+ 7763us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:53.089688 (+1088543us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:53.289675 (+199987us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540067d9d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:53.289912 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2448348032}
I20251023 02:04:53.348578 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:53.349114 23375 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:53.350390 23005 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.365614 22000 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:53.350319 22002 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.351670 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.354439 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.353271 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.357230 22034 raft_consensus.cc:1240] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:53.365994 22110 raft_consensus.cc:1240] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:53.367221 22111 raft_consensus.cc:1240] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:53.372965 22002 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:53.339951 22032 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:04:53.355722 23271 raft_consensus.cc:740] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:53.350337 23405 consensus_queue.cc:1048] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20251023 02:04:53.387679 21859 catalog_manager.cc:5649] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 3, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:04:53.389755 21859 catalog_manager.cc:5649] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 3, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 3 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:53.391532 21859 catalog_manager.cc:5649] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:53.394666 23214 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:53.394948 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.395656 22104 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:53.396061 22108 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:53.396399 23464 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:53.396934 23464 raft_consensus.cc:697] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:53.397786 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.397925 23464 consensus_queue.cc:237] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.399978 23297 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:53.400545 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.401407 22029 raft_consensus.cc:2393] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:53.402005 23428 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:53.402475 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.403952 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.404821 22030 raft_consensus.cc:2393] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:53.400596 23297 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.407517 22106 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:04:53.408499 23297 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.409387 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.410109 22031 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:53.379360 23377 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:53.412211 23404 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.385110 23407 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:53.414733 23407 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:53.415191 23407 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:53.415670 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 113896us
I20251023 02:04:53.401232 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.416878 22035 raft_consensus.cc:2393] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:53.419394 22590 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:53.425868 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.393252 21859 catalog_manager.cc:5649] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 2, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 2 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:04:53.431612 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.432386 21960 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:53.429972 21859 catalog_manager.cc:5649] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:04:53.437870 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:53.396950 22102 raft_consensus.cc:2410] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:04:53.379724 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.443380 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:53.444044 21957 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.440936 22034 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:53.447911 21859 catalog_manager.cc:5649] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 2, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 2 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:53.370147 23433 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.397814 23439 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.435696 23405 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:53.436491 23428 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:53.440905 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:53.443957 22076 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.696s (SENT) [suppressed 30 similar messages]
W20251023 02:04:53.473678 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 44794.6us
I20251023 02:04:53.474287 23379 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:53.477399 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 96260.9us
I20251023 02:04:53.475311 23404 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.476246 23271 consensus_queue.cc:260] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.444991 23440 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:53.470157 23405 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:53.447031 23377 raft_consensus.cc:697] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:53.477402 22105 raft_consensus.cc:2368] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5.
I20251023 02:04:53.485015 21957 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:53.477545 23428 raft_consensus.cc:697] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:53.486330 23405 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:53.487444 23005 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:53.493098 23271 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.513933 23319 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:53.521826 23377 consensus_queue.cc:237] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.522233 23319 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.491284 23214 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:53.528770 21928 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:53.528965 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 170648us
I20251023 02:04:53.537221 22002 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:53.493403 23440 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:53.545270 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 101364us
W20251023 02:04:53.546473 23469 log.cc:927] Time spent T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.132s user 0.003s sys 0.000s
I20251023 02:04:53.552646 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:53.553354 22106 raft_consensus.cc:2393] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:53.552405 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:53.535035 23375 raft_consensus.cc:697] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:53.554116 22111 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.546528 23465 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:53.557183 23465 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.558044 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.558820 22106 raft_consensus.cc:2393] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:53.559173 23465 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.549468 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:53.562582 22104 raft_consensus.cc:2393] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:53.551662 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:53.563565 21955 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.568112 23378 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.555629 22619 raft_consensus.cc:1240] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:53.570168 23433 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:53.571782 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 50596.8us
I20251023 02:04:53.572540 21928 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.549942 23297 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:53.578977 23428 consensus_queue.cc:237] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.579480 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:53.580155 22108 raft_consensus.cc:2393] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:53.581046 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.536211 23005 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.581676 22104 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
W20251023 02:04:53.538775 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 92057.7us
I20251023 02:04:53.584328 22111 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:53.586848 21928 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:53.555931 23456 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20251023 02:04:53.590636 23433 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:53.591408 23433 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:53.590538 21925 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:53.552551 23319 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:53.598492 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 61321.9us
I20251023 02:04:53.600767 23297 raft_consensus.cc:2764] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20251023 02:04:53.603615 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 84121.3us
I20251023 02:04:53.577652 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:53.605739 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.607613 23216 raft_consensus.cc:1064] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:53.607807 21925 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.608006 23478 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:53.607606 23297 consensus_queue.cc:1059] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:04:53.612715 22002 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:53.582824 23440 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.607973 23309 raft_consensus.cc:1064] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:04:53.571063 23214 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.622014 23477 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:53.622247 23297 raft_consensus.cc:3055] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:53.622520 23477 raft_consensus.cc:697] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:53.622735 23297 raft_consensus.cc:740] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:53.623553 23477 consensus_queue.cc:237] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.623667 23297 consensus_queue.cc:260] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.625022 23297 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.625427 23473 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:53.625743 23444 consensus_queue.cc:1059] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:53.625876 21925 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.628840 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.641261 22109 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:53.625960 23473 raft_consensus.cc:697] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:53.643560 21955 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:53.618058 22103 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.618220 23478 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:53.627409 23444 raft_consensus.cc:3055] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:53.648514 23444 raft_consensus.cc:740] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:53.649405 23444 consensus_queue.cc:260] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.650785 23444 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.620342 23005 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.657289 22111 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251023 02:04:53.640662 23378 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:53.658771 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 54047.3us
I20251023 02:04:53.628161 23375 consensus_queue.cc:237] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.651119 23407 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.651701 23440 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:53.662693 23216 consensus_queue.cc:237] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.665084 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:53.678808 22111 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:53.668172 22076 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:53.661935 23214 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:53.676374 21925 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.665901 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.651945 23478 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:53.700712 21928 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.682313 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:53.678861 22109 raft_consensus.cc:2376] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4. Re-sending same reply.
I20251023 02:04:53.702328 21957 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.644654 23473 consensus_queue.cc:237] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.705195 22002 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:53.712388 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:53.723783 23480 log.cc:927] Time spent T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20251023 02:04:53.723923 23476 log.cc:927] Time spent T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20251023 02:04:53.724567 23405 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:53.728348 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 234319us
I20251023 02:04:53.659085 23433 consensus_queue.cc:1059] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20251023 02:04:53.729632 23405 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.734874 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:53.711856 23378 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:53.672125 23444 consensus_queue.cc:1059] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
W20251023 02:04:53.739825 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:53.681952 23437 consensus_queue.cc:1059] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
W20251023 02:04:53.746348 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 70105us
I20251023 02:04:53.743155 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.726756 23214 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:53.732878 23271 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:53.749315 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 81263.5us
I20251023 02:04:53.750242 21925 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.733114 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.755825 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.761075 21957 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:53.745813 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:53.732316 23377 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:53.722254 23440 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:53.750746 22102 raft_consensus.cc:2393] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:53.766934 21955 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:53.773975 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.774781 22037 raft_consensus.cc:2393] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:53.775628 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.749913 23404 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:53.780314 23404 raft_consensus.cc:3055] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:53.780699 23404 raft_consensus.cc:740] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:53.781622 23404 consensus_queue.cc:260] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.764865 22031 raft_consensus.cc:2393] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:53.783013 23404 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:53.754215 23378 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.788075 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.788743 22032 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:53.789251 23407 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.789496 23378 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:53.789616 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.754426 23433 consensus_queue.cc:1243] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer 1ab8b0b58147446d84d901b4ed2ff39e log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251023 02:04:53.767210 23214 raft_consensus.cc:697] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:53.790923 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.791679 23214 consensus_queue.cc:237] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.792181 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:53.766716 23271 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.795550 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.795859 23271 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:53.796738 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.797700 23440 raft_consensus.cc:3055] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:53.797948 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.801138 23377 raft_consensus.cc:697] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:53.787770 21954 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:53.802140 23377 consensus_queue.cc:237] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:53.802577 23491 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:53.803148 23491 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:53.803568 23491 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.798080 23440 raft_consensus.cc:740] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:53.783396 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.802621 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.792872 22032 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:53.821847 22000 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.827998 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:53.828763 22035 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:53.824158 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.829943 22106 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
W20251023 02:04:53.834729 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 56404.9us
W20251023 02:04:53.837483 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 136920us
I20251023 02:04:53.827150 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:53.848397 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 98242.3us
I20251023 02:04:53.840772 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:53.850317 22106 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.850605 21925 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:53.852062 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 104935us
I20251023 02:04:53.839632 21928 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.861106 21925 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.853653 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.854292 22031 raft_consensus.cc:2393] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:53.836968 23477 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:53.870963 22002 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:53.820973 23430 consensus_queue.cc:1059] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20251023 02:04:53.868140 22036 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:53.874416 22002 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:53.878316 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:53.879026 22111 raft_consensus.cc:2393] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:53.887470 22106 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:53.854108 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:53.889688 22002 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:53.892901 22034 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:53.894958 22033 raft_consensus.cc:1240] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
W20251023 02:04:53.899749 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 55056us
I20251023 02:04:53.903177 22035 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:04:53.906998 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:53.907728 22106 raft_consensus.cc:2393] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:53.891031 23005 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:53.894214 23196 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:53.892900 22029 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251023 02:04:53.890609 23430 consensus_queue.cc:1243] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20251023 02:04:53.922503 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 61478.7us
I20251023 02:04:53.912045 23494 consensus_queue.cc:1059] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251023 02:04:53.917089 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 112625us
I20251023 02:04:53.897483 23437 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:53.928272 22000 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:53.930210 22077 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:53.930914 22103 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:53.931320 23430 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:53.931756 23430 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:53.932090 23430 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:53.921772 23196 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:53.933912 23196 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:53.927068 22106 raft_consensus.cc:1275] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251023 02:04:53.925441 22035 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20251023 02:04:53.925858 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 86335.2us
I20251023 02:04:53.899608 23465 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:53.941707 22076 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:53.942440 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:53.903709 23119 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
W20251023 02:04:53.997275 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 53366us
I20251023 02:04:53.997933 21925 leader_election.cc:400] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5.
I20251023 02:04:53.998365 21925 leader_election.cc:403] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:04:53.998287 22000 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:53.895110 23440 consensus_queue.cc:260] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.906880 23493 consensus_queue.cc:1059] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20251023 02:04:53.939297 23465 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:53.939129 23005 raft_consensus.cc:697] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:54.004235 22111 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
W20251023 02:04:54.004802 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 61708.5us
W20251023 02:04:54.001936 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 60315.2us
I20251023 02:04:53.944130 23437 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:53.943769 23483 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:54.008728 22000 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:54.007148 23405 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:54.009943 23479 consensus_queue.cc:1048] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:54.010372 23405 raft_consensus.cc:697] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:54.011334 23440 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:54.011456 23405 consensus_queue.cc:237] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.012043 23437 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:54.005895 23297 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:54.013597 23437 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:53.944375 23428 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.070s
I20251023 02:04:54.014937 22076 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.016041 22076 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.016996 23499 raft_consensus.cc:2764] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20251023 02:04:53.944293 23309 raft_consensus.cc:1068] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: Unable to promote non-voter 1ab8b0b58147446d84d901b4ed2ff39e: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:53.992465 23456 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.025s
I20251023 02:04:54.018482 22076 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.003764 23216 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:54.019671 23216 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:54.021001 22076 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.021718 23216 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:54.023171 23005 consensus_queue.cc:237] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.026160 22076 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.026885 23216 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:54.027292 23433 raft_consensus.cc:3055] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:54.027688 23433 raft_consensus.cc:740] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:54.028719 23428 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.028450 23433 consensus_queue.cc:260] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.029078 23216 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:54.029206 23428 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:54.045766 23428 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:54.046051 22106 raft_consensus.cc:1275] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20251023 02:04:54.046234 23166 raft_consensus.cc:993] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: : Instructing follower 02c53ba06cc3445a8534a84aa76b1c14 to start an election
I20251023 02:04:54.046394 22106 pending_rounds.cc:85] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 1
I20251023 02:04:54.046746 22106 pending_rounds.cc:107] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251023 02:04:54.038381 22035 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251023 02:04:54.046979 22106 raft_consensus.cc:2889] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:04:54.040223 23493 raft_consensus.cc:3055] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:54.041518 23494 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 5
W20251023 02:04:54.041915 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 99299.6us
I20251023 02:04:54.007608 23465 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:04:54.043141 23502 consensus_queue.cc:1175] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "818907d63f1f4d84ba267ebe6ee837e0" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251023 02:04:54.050047 21928 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:54.050746 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.051474 21957 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:54.030675 22076 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:04:54.058614 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:54.059676 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:54.061107 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:54.066254 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:54.029570 23433 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:54.073036 22077 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:54.073294 22108 raft_consensus.cc:1240] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
W20251023 02:04:54.074981 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 44380.1us
I20251023 02:04:54.076915 22109 raft_consensus.cc:1240] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:54.076434 23497 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:54.048673 23493 raft_consensus.cc:740] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:54.090641 21958 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251023 02:04:54.073961 22028 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:54.093379 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:54.093917 21953 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:54.094092 21955 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
W20251023 02:04:54.094635 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 46220.3us
I20251023 02:04:54.090179 23499 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:54.089635 23378 raft_consensus.cc:1081] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Signalling peer 02c53ba06cc3445a8534a84aa76b1c14 to start an election
I20251023 02:04:54.109989 21958 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:54.076610 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:54.077231 22077 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:54.098279 23406 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:54.116408 21928 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.118582 21928 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.114512 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.098585 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:54.123054 23444 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:54.123198 22103 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:04:54.106518 23166 consensus_queue.cc:1048] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
W20251023 02:04:54.125921 23498 log.cc:927] Time spent T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.112s user 0.002s sys 0.000s
I20251023 02:04:54.126382 23491 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.127856 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:54.128434 23491 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:54.128556 21955 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:54.102874 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:54.139103 21957 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:04:54.139983 22589 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
I20251023 02:04:54.122109 22077 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:54.140308 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.142419 21925 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:54.145509 21928 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:54.115482 21954 raft_consensus.cc:1275] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251023 02:04:54.147251 23501 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:54.147773 23501 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:54.148203 23501 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:54.129469 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.150856 21928 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:54.120162 23216 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:54.119735 23500 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:54.144662 23491 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:54.151424 23499 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:04:54.113240 23437 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.155553 23479 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.138463 23119 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20251023 02:04:54.155999 23216 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:54.157938 23500 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:54.145915 23297 consensus_queue.cc:1059] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20251023 02:04:54.143991 23464 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20251023 02:04:54.151362 23428 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20251023 02:04:54.212194 23479 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:54.206573 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.206885 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.207949 22103 raft_consensus.cc:1275] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:54.208194 23497 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:54.198066 23309 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20251023 02:04:54.198405 23430 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:54.216660 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 128432us
I20251023 02:04:54.194177 22031 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:04:54.202225 21955 raft_consensus.cc:2393] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:54.209368 23465 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.202059 23491 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:54.204908 23216 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:54.193570 23196 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.251371 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:54.214536 23500 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:54.216869 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.254667 22110 raft_consensus.cc:2393] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:54.238883 23497 raft_consensus.cc:697] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:54.263954 23497 consensus_queue.cc:237] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:54.274060 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 131729us
I20251023 02:04:54.201651 23377 consensus_queue.cc:1048] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20251023 02:04:54.238492 23508 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.143802 23493 consensus_queue.cc:260] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.258677 23467 raft_consensus.cc:2764] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20251023 02:04:54.257905 23473 raft_consensus.cc:3055] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:54.269519 23437 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:54.270017 23479 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:54.362627 22590 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:04:54.365835 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 215090us
I20251023 02:04:54.299403 22028 raft_consensus.cc:1240] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
I20251023 02:04:54.299870 23473 raft_consensus.cc:740] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:54.387830 21961 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
W20251023 02:04:54.390148 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 109026us
I20251023 02:04:54.365556 23513 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.311026 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.404628 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.405532 21961 raft_consensus.cc:2368] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
W20251023 02:04:54.408536 23478 consensus_meta.cc:287] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.682s user 0.010s sys 0.000s
I20251023 02:04:54.408468 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:54.409024 23478 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.410457 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:54.410928 23478 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:54.387642 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:54.412400 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:54.414898 21961 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:54.300410 23444 consensus_queue.cc:1059] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.098s
I20251023 02:04:54.420992 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:54.422369 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:54.407546 23119 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:54.428403 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.434971 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:54.435717 22035 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 5
W20251023 02:04:54.436198 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:54.298588 23508 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:54.432926 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:54.389799 23385 consensus_queue.cc:1048] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20251023 02:04:54.403499 23499 raft_consensus.cc:697] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:54.441892 21928 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:54.444437 22002 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:54.455662 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:54.456770 22034 raft_consensus.cc:2393] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:54.469964 21961 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:54.438334 23119 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:54.439669 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:54.403988 23513 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:54.475364 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 112884us
I20251023 02:04:54.482993 23196 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:54.446965 23430 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:54.492585 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:54.495175 21961 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251023 02:04:54.463701 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:04:54.513607 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:54.524179 22107 raft_consensus.cc:1275] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:54.475157 23005 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:54.472725 23502 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:54.483278 23119 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:54.476809 23465 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:54.484838 22031 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251023 02:04:54.535279 23005 raft_consensus.cc:697] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:54.475550 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.537698 23524 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:54.537786 21955 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:54.538127 23005 consensus_queue.cc:237] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.538308 23524 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.540536 23524 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:54.434274 23473 consensus_queue.cc:260] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.541216 23502 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.545125 23531 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:04:54.546842 23502 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:54.548055 23428 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251023 02:04:54.548395 23518 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:54.548754 23440 consensus_meta.cc:287] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.537s user 0.007s sys 0.000s
I20251023 02:04:54.548964 23518 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:54.549633 23510 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.249s user 0.000s sys 0.001s
I20251023 02:04:54.551306 23518 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:54.551790 23508 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:04:54.552446 23526 log.cc:927] Time spent T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20251023 02:04:54.553035 23485 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:54.553705 22619 raft_consensus.cc:2393] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:54.553601 23485 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:54.555564 23485 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:54.555320 23216 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
W20251023 02:04:54.556034 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 73848.1us
I20251023 02:04:54.556424 21955 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:54.536109 22032 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:54.559799 22001 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:54.512243 22035 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:04:54.494678 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:54.584753 22110 raft_consensus.cc:2393] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:54.493914 21954 raft_consensus.cc:2393] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:54.532193 23483 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:54.637161 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 77441.5us
W20251023 02:04:54.637387 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 195558us
I20251023 02:04:54.633488 22110 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:54.633417 22107 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251023 02:04:54.639122 22076 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.639881 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:54.640646 22107 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
W20251023 02:04:54.636461 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 76679.7us
I20251023 02:04:54.557427 23216 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:54.544126 23491 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251023 02:04:54.650548 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 206254us
I20251023 02:04:54.650646 21928 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:54.641961 21925 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:54.643352 22107 raft_consensus.cc:1275] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:54.644165 23478 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:54.645567 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:54.657961 22108 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:54.646716 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:54.646770 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:54.647701 22076 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.658792 22106 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:54.641675 23430 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.646131 23465 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.659874 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:54.654767 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:54.670552 22110 raft_consensus.cc:2393] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:54.632957 21961 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b"
dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:53734
I20251023 02:04:54.654261 23440 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:54.671123 21961 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:04:54.671471 21961 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:54.654518 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:54.662254 22076 leader_election.cc:400] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
I20251023 02:04:54.663005 23522 raft_consensus.cc:2749] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251023 02:04:54.663270 23517 log.cc:927] Time spent T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.255s user 0.001s sys 0.000s
I20251023 02:04:54.663594 23525 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.683660 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.666852 23406 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:54.684317 21954 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:54.684710 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:54.685313 22029 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:54.661461 23502 consensus_queue.cc:1048] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20251023 02:04:54.661361 23309 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:54.663601 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:54.654029 23518 raft_consensus.cc:2749] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:04:54.675422 22076 leader_election.cc:403] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251023 02:04:54.722205 22103 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:04:54.726362 23513 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:54.684414 23406 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:54.656106 23479 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:54.680832 22109 raft_consensus.cc:1240] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:54.683454 23525 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.664131 23523 consensus_queue.cc:1048] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20251023 02:04:54.667097 23500 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.679273 21961 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:54.732898 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 48727.7us
I20251023 02:04:54.698143 22108 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:54.672369 22002 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:54.685427 23501 consensus_meta.cc:287] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.537s user 0.008s sys 0.000s
I20251023 02:04:54.557682 22587 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:54.669773 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:54.677692 21928 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:04:54.740346 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 86131.9us
I20251023 02:04:54.675117 23440 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:54.675673 21953 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:54.749274 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:54.648764 23483 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.671797 23430 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:54.735241 21961 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:54.744596 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:54.756695 22034 raft_consensus.cc:3055] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:54.757102 22034 raft_consensus.cc:740] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:54.748144 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:54.674896 23465 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.749977 22036 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:04:54.731978 22106 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:04:54.729338 21955 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20251023 02:04:54.758381 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 99084.2us
I20251023 02:04:54.772749 22076 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:04:54.748911 23512 log.cc:927] Time spent T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.344s user 0.001s sys 0.000s
I20251023 02:04:54.757932 22034 consensus_queue.cc:260] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.777027 22034 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:54.778424 23537 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election won for term 5
W20251023 02:04:54.779160 23504 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.543s user 0.001s sys 0.000s
I20251023 02:04:54.779353 23214 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:54.779798 23214 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:54.780136 23214 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:54.750597 23440 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:04:54.791517 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 117732us
I20251023 02:04:54.781867 23537 raft_consensus.cc:697] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:54.806488 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:54.810667 21961 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:54.654975 23499 consensus_queue.cc:237] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.770207 23309 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3.
W20251023 02:04:54.855549 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:54.740092 23479 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:54.894392 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 121787us
I20251023 02:04:54.894774 21928 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:54.826434 22104 raft_consensus.cc:2393] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:54.752101 23501 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.813166 23522 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.905018 22076 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.912134 22031 raft_consensus.cc:2393] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:54.897833 23196 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:54.932718 22034 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:04:54.906353 23465 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:54.938581 22076 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.913007 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:54.938040 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 265755us
I20251023 02:04:54.938308 23548 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:54.944885 22075 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.943416 22076 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.946444 22076 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.947943 21928 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:54.949316 22076 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:54.935772 23309 raft_consensus.cc:2749] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:04:54.935053 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:54.946141 23494 consensus_meta.cc:287] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.876s user 0.010s sys 0.002s
I20251023 02:04:54.940016 23430 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:54.953056 22002 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:54.939713 23196 raft_consensus.cc:697] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:04:54.951773 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 247683us
I20251023 02:04:54.947552 23502 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.947985 21953 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:54.958541 23525 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:54.943367 23543 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:54.966385 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:54.966151 23537 consensus_queue.cc:237] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.966657 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:54.988718 21959 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:54.966193 23532 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251023 02:04:54.946906 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 389366us
W20251023 02:04:54.967819 23539 log.cc:927] Time spent T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
W20251023 02:04:54.968648 23545 log.cc:927] Time spent T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20251023 02:04:54.962615 23530 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.921902 23522 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:54.970162 23196 consensus_queue.cc:237] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:54.957047 23433 consensus_meta.cc:287] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.865s user 0.009s sys 0.003s
I20251023 02:04:54.959846 23494 raft_consensus.cc:2749] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5.
I20251023 02:04:54.949076 23524 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251023 02:04:54.977670 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:55.005347 21925 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:54.953483 23513 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:54.955011 23216 raft_consensus.cc:2804] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:04:54.969255 23554 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:54.990550 23532 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:54.998771 21955 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251023 02:04:54.978248 23465 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:54.984687 23502 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:54.997879 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 49635.7us
I20251023 02:04:55.002867 22002 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:54.956532 23542 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:04:55.004088 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 48795.9us
I20251023 02:04:54.961129 23508 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20251023 02:04:54.992856 23525 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.992794 22104 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:55.032238 21957 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:55.003779 23483 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:54.992136 23500 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:55.015260 23513 raft_consensus.cc:697] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:55.021045 22032 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:55.021436 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:55.041507 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:55.042495 21961 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:55.043754 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:55.017807 23465 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:55.015444 23532 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:55.022176 23554 raft_consensus.cc:697] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:55.023454 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:55.024080 23502 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:55.025076 21959 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:55.025875 23216 raft_consensus.cc:697] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:55.026237 23547 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:55.026788 23319 raft_consensus.cc:3055] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
W20251023 02:04:55.028703 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:55.027383 22029 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:55.039028 23479 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:55.060882 23522 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:55.060514 23216 consensus_queue.cc:237] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.045126 23501 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:55.034389 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:55.015522 23433 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:55.033352 22105 raft_consensus.cc:1275] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:55.046005 22076 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:55.047196 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:55.047926 21961 raft_consensus.cc:2393] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:54.999138 23530 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.074021 22076 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:55.072104 21958 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:04:55.051102 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:55.076139 21959 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:55.077052 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:55.048004 23465 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:55.045351 23430 consensus_queue.cc:1059] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20251023 02:04:55.037067 23525 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:55.009083 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 61227.1us
I20251023 02:04:55.079113 22002 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:55.081465 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:55.058264 21955 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:55.065896 21954 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:55.056335 23319 raft_consensus.cc:740] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:55.061087 23406 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.095346 21959 raft_consensus.cc:2410] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251023 02:04:55.104453 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:55.079695 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:55.105113 22621 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:55.128132 21961 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:55.080360 23522 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:55.096201 22029 raft_consensus.cc:1275] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:55.129248 22002 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:55.130074 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:55.130084 23560 log.cc:927] Time spent T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20251023 02:04:55.132125 23501 consensus_queue.cc:1048] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:55.132409 21928 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:55.133536 23502 raft_consensus.cc:2749] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:04:55.135335 23405 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.115s user 0.010s sys 0.006s
W20251023 02:04:55.135790 23405 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.116s user 0.010s sys 0.006s
I20251023 02:04:55.078727 23500 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:55.100208 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:55.146518 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 111207us
I20251023 02:04:55.146696 22107 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:04:55.094723 23433 raft_consensus.cc:697] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:55.151374 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:55.156015 21925 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:55.148892 22076 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:55.157603 22035 raft_consensus.cc:1240] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:55.084652 23430 consensus_queue.cc:1243] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251023 02:04:55.126753 23407 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:55.079845 23119 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.139021 21953 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:04:55.186942 23407 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:55.188278 23430 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:55.188789 23494 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:55.153424 23522 raft_consensus.cc:697] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:55.141115 23499 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:55.096082 22587 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:55.149792 23352 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:55.191303 21961 raft_consensus.cc:1240] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251023 02:04:55.200248 23499 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.198736 23404 raft_consensus.cc:3055] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:55.202122 23404 raft_consensus.cc:740] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:55.200248 23352 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.203123 23557 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:55.192147 22619 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEARNER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251023 02:04:55.190073 23563 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:55.192467 23407 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:55.140765 23500 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:55.160606 23524 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:55.102357 23531 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20251023 02:04:55.155572 23502 consensus_queue.cc:1048] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20251023 02:04:55.193800 23430 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:55.197897 23478 consensus_meta.cc:287] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.537s user 0.010s sys 0.000s
I20251023 02:04:55.198318 23522 consensus_queue.cc:237] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:55.206276 23563 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:55.204381 22002 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:55.164952 23508 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:55.095717 23550 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.126s
I20251023 02:04:55.095626 23555 consensus_queue.cc:1059] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.185s
I20251023 02:04:55.216468 23524 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.218467 23530 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:55.223198 23352 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:55.223335 23553 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:55.223621 21953 raft_consensus.cc:1275] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:55.223860 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:55.244508 23553 raft_consensus.cc:697] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:55.244705 22034 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:55.224462 23499 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:04:55.225020 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 129072us
I20251023 02:04:55.246498 22590 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:55.226511 23502 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:55.198182 23494 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.210219 23500 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:55.227351 23478 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:55.239063 23542 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:04:55.153851 23554 consensus_queue.cc:237] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.243630 23541 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:04:55.165220 23513 consensus_queue.cc:237] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.235059 23563 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:55.259083 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 110286us
I20251023 02:04:55.235357 23508 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:55.237706 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53732 (request call id 936) took 1370 ms. Trace:
I20251023 02:04:55.267238 21956 rpcz_store.cc:276] 1023 02:04:53.861968 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:53.865731 (+ 3763us) service_pool.cc:225] Handling call
1023 02:04:55.224569 (+1358838us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b5800002a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9ae20e 000000000035a3c9 00007f9c649cd030 00007f9c64a957f1 00007f9c64a959b0 00007f9c64a46f02 00007f9c5dfee437 00007f9c5dfee3c2 00007f9c5dfee2f1
1023 02:04:55.229795 (+ 5226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6593920}
I20251023 02:04:55.245857 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:55.236178 23216 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:55.245407 23553 consensus_queue.cc:237] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.258944 23530 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:55.273862 22076 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:55.259332 23405 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:55.265246 23406 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:55.297112 23525 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:55.278553 23530 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:55.297967 23530 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:04:55.300097 23214 consensus_meta.cc:287] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.520s user 0.008s sys 0.000s
I20251023 02:04:55.300498 23214 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.283638 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:55.271351 23508 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:55.278354 23216 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.259982 23433 consensus_queue.cc:237] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.329628 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:55.266189 23557 raft_consensus.cc:697] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:55.333359 22034 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:55.258666 23502 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.303964 23271 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:04:55.332288 23478 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:55.336412 23406 consensus_queue.cc:1048] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251023 02:04:55.278215 23541 raft_consensus.cc:697] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:55.301491 22104 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:04:55.357743 23572 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:55.297173 23555 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:55.301972 23548 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:55.358284 23271 raft_consensus.cc:697] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:55.377816 23502 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:55.304195 23405 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.398916 23405 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:55.383179 23440 consensus_meta.cc:287] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.559s user 0.010s sys 0.000s
I20251023 02:04:55.279031 23531 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.115s
W20251023 02:04:55.401513 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 111975us
I20251023 02:04:55.402498 22034 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:04:55.403402 23575 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:55.404150 23575 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.404834 22590 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:55.405505 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:55.360332 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:55.346230 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 71157.1us
I20251023 02:04:55.359192 23119 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:55.359717 23558 raft_consensus.cc:2955] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:55.360325 23478 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.381500 23530 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.377087 23319 consensus_queue.cc:260] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.400472 23532 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.401018 23555 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.375098 23557 consensus_queue.cc:237] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:55.471892 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:55.487586 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:55.521449 21962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1187) took 1657 ms (client timeout 2122 ms). Trace:
W20251023 02:04:55.528113 21962 rpcz_store.cc:269] 1023 02:04:53.863834 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:53.868186 (+ 4352us) service_pool.cc:225] Handling call
1023 02:04:55.487397 (+1619211us) spinlock_profiling.cc:243] Waited 5.14 ms on lock 0x7b5800002a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:04:55.521393 (+ 33996us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11818368}
W20251023 02:04:55.502590 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 245439us
W20251023 02:04:55.530884 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:55.518159 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 67521.4us
I20251023 02:04:55.408866 23440 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.412667 23525 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:04:55.533712 23570 log.cc:927] Time spent T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20251023 02:04:55.480967 23404 consensus_queue.cc:260] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.534409 22105 raft_consensus.cc:1240] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:55.540230 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:55.557371 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:55.558089 22108 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:55.411679 23548 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader election won for term 5
W20251023 02:04:55.533568 23506 log.cc:927] Time spent T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 1.208s user 0.004s sys 0.026s
I20251023 02:04:55.449872 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:55.555591 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:55.500506 23494 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:55.520859 23214 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:55.476447 23577 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:55.561543 22111 raft_consensus.cc:1275] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:55.561864 22036 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:55.547777 23525 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:55.566112 23581 log.cc:927] Time spent T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20251023 02:04:55.566475 23521 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:55.569681 21962 raft_consensus.cc:2393] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:55.443210 23465 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:55.552376 23319 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:55.552482 23467 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:55.566634 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:55.478982 23574 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:55.598273 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:55.478400 23500 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.544174 23563 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.792591 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:55.410055 23572 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:55.573658 23548 raft_consensus.cc:697] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:55.487140 23271 consensus_queue.cc:237] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.828512 23576 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:55.412810 23541 consensus_queue.cc:237] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.848311 21962 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:55.862465 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 931) took 1226 ms. Trace:
I20251023 02:04:55.862846 21962 rpcz_store.cc:276] 1023 02:04:54.635711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.836791 (+1201080us) service_pool.cc:225] Handling call
1023 02:04:55.848069 (+ 11278us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:55.862156 (+ 14087us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:55.862397 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:55.850682 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 280885us
I20251023 02:04:55.858711 22108 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:55.790131 23309 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:55.866051 23576 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.898864 23576 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:55.899529 23576 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:55.900012 23576 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.804854 23513 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:04:55.845762 23573 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20251023 02:04:55.906401 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:55.829525 22103 raft_consensus.cc:1240] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:55.906893 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:55.816812 23494 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:55.829211 23540 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:55.852655 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:55.839715 23214 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:55.851037 23478 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:55.907853 23407 consensus_meta.cc:287] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.692s user 0.005s sys 0.005s
I20251023 02:04:55.845970 23577 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.901825 23309 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.930194 23575 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:55.934592 23530 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:55.912436 23352 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:55.947664 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:55.917430 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:55.842918 23521 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.068044 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.064998 22030 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:56.068795 22036 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:55.937608 22623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34806 (request call id 272) took 1333 ms. Trace:
I20251023 02:04:55.935905 22105 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:55.912935 23513 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.067654 22103 raft_consensus.cc:1275] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:56.076840 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:56.062757 23406 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:56.062273 23532 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:55.923327 23494 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.093153 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:55.943380 23407 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.074659 22623 rpcz_store.cc:276] 1023 02:04:54.585859 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:54.636300 (+ 50441us) service_pool.cc:225] Handling call
1023 02:04:55.566464 (+930164us) spinlock_profiling.cc:243] Waited 4.42 ms on lock 0x7b5800214598. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:04:55.919365 (+352901us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10160512}
I20251023 02:04:55.936499 23214 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:56.071715 22032 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:56.071187 23555 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:55.886811 23467 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.907677 23574 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:55.935058 23548 consensus_queue.cc:237] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:55.945118 23531 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:56.065701 23352 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:56.080062 23500 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.088266 23525 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.091881 23440 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:56.096552 23572 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.113698 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.113878 22033 raft_consensus.cc:2368] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5.
I20251023 02:04:56.098639 23406 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.107479 23532 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:56.066123 23540 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.111884 23563 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.114411 23465 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.142979 23574 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.113528 23576 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.137678 23352 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:56.121956 23494 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.139458 23532 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:56.156278 22107 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.761s user 0.045s sys 0.110s
W20251023 02:04:56.156735 22107 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s user 0.045s sys 0.110s
I20251023 02:04:56.157039 23532 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:56.160818 23553 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.001s sys 0.001s
W20251023 02:04:56.161257 23553 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s user 0.001s sys 0.001s
W20251023 02:04:56.196692 21925 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.002s sys 0.054s
W20251023 02:04:56.197186 21925 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.659s user 0.002s sys 0.054s
I20251023 02:04:56.122822 23531 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:56.236408 23521 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.238788 23319 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:56.207756 23555 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:56.205159 23309 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.209316 23532 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:56.239355 23319 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.257138 23577 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:56.273299 22001 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.688s user 0.009s sys 0.011s
W20251023 02:04:56.273633 22001 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.689s user 0.010s sys 0.011s
I20251023 02:04:56.265004 23407 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:56.277838 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 408610us
I20251023 02:04:56.267556 23513 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.257391 23555 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.288481 23406 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.274273 23531 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:56.288671 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:56.289234 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:56.273218 23214 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:56.259836 23532 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:56.331568 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:56.292860 22036 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
W20251023 02:04:56.318046 21954 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.014s sys 0.030s
I20251023 02:04:56.317884 23440 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:56.335029 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:56.343182 22077 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:56.351168 22077 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:04:56.323494 23522 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.004s sys 0.018s
I20251023 02:04:56.321424 23352 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:04:56.334996 21954 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.854s user 0.014s sys 0.030s
W20251023 02:04:56.338944 23579 log.cc:927] Time spent T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20251023 02:04:56.330404 23319 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.328533 23513 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:04:56.406136 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 55047.2us
I20251023 02:04:56.337579 23467 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.411536 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.412122 22077 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:56.412854 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:56.395540 23522 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.831s user 0.005s sys 0.019s
I20251023 02:04:56.411237 23309 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:56.340611 23406 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:56.341549 23540 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.339322 23578 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:56.413172 23467 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:04:56.424310 22077 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:56.433455 23309 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:56.414565 23374 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:56.417431 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.398787 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:56.401643 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:56.413518 22030 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:56.418051 22029 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:04:56.437556 21928 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.024s sys 0.032s
I20251023 02:04:56.396893 23214 consensus_queue.cc:1059] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20251023 02:04:56.420428 23574 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:56.452250 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1211) took 1214 ms. Trace:
I20251023 02:04:56.434124 23500 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:04:56.456463 22107 rpcz_store.cc:276] 1023 02:04:55.237757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.237874 (+ 117us) service_pool.cc:225] Handling call
1023 02:04:55.238291 (+ 417us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:04:55.260769 (+ 22478us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:55.260816 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:04:55.266762 (+ 5946us) log.cc:844] Serialized 52 byte log entry
1023 02:04:56.157187 (+890425us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:56.450581 (+293394us) log.cc:844] Serialized 14 byte log entry
1023 02:04:56.451926 (+ 1345us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:56.452002 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:56.452038 (+ 36us) raft_consensus.cc:1685] finished
1023 02:04:56.452055 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:56.452191 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":380,"raft.queue_time_us":12361,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":7286,"thread_start_us":1049365,"threads_started":2,"wal-append.queue_time_us":853985}
I20251023 02:04:56.460649 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.461514 22077 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:56.443195 23309 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:56.461328 22030 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:56.443921 23319 raft_consensus.cc:2749] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:04:56.456877 22003 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.852s user 0.008s sys 0.029s
I20251023 02:04:56.407651 23352 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.418123 23440 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:56.458849 23508 consensus_meta.cc:287] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.122s user 0.011s sys 0.000s
W20251023 02:04:56.470857 22003 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.866s user 0.009s sys 0.030s
W20251023 02:04:56.473503 21928 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.029s sys 0.033s
I20251023 02:04:56.451428 22105 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:56.479184 22077 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:56.438357 23513 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.487120 23522 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:56.489689 23524 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:56.489903 23500 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:56.510421 23500 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:56.510829 23500 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:56.496431 23430 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:56.512924 23430 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:56.513249 23430 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:56.505213 23508 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.490221 23532 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.443961 23578 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:56.494441 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:56.494938 23555 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.470445 23407 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:56.495651 23352 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.441179 23406 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:56.464862 23214 consensus_queue.cc:1243] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer 818907d63f1f4d84ba267ebe6ee837e0 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251023 02:04:56.444310 23374 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:56.489401 23558 consensus_meta.cc:287] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.604s user 0.010s sys 0.000s
I20251023 02:04:56.516502 23508 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.516865 23522 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.519156 23513 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.513561 23430 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:56.545310 23522 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.551256 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.552775 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.521204 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1206) took 1556 ms. Trace:
W20251023 02:04:56.522881 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:56.460908 23574 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.543313 23406 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:56.513967 23540 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:56.531563 22107 raft_consensus.cc:2955] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:56.609532 22029 raft_consensus.cc:1275] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:04:56.610018 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:56.611169 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 97282.9us
I20251023 02:04:56.611272 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:56.617353 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.567963 22106 rpcz_store.cc:276] 1023 02:04:54.965064 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:54.965172 (+ 108us) service_pool.cc:225] Handling call
1023 02:04:54.965573 (+ 401us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:55.012279 (+ 46706us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:55.012319 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:55.012848 (+ 529us) log.cc:844] Serialized 26 byte log entry
1023 02:04:55.116145 (+103297us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:55.116203 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:56.520764 (+1404561us) spinlock_profiling.cc:243] Waited 46.2 ms on lock 0x7b54001ac2d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:56.520825 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:56.520888 (+ 63us) raft_consensus.cc:1685] finished
1023 02:04:56.520906 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:56.521139 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":131,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":4131,"spinlock_wait_cycles":106334464,"thread_start_us":63208,"threads_started":1,"wal-append.queue_time_us":174071,"wal-append.run_cpu_time_us":1775,"wal-append.run_wall_time_us":1030069}
I20251023 02:04:56.630194 21928 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:56.641384 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1204) took 1989 ms. Trace:
I20251023 02:04:56.665537 21962 rpcz_store.cc:276] 1023 02:04:54.651800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.896221 (+1244421us) service_pool.cc:225] Handling call
1023 02:04:55.913114 (+ 16893us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:55.919811 (+ 6697us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:55.919854 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:55.944667 (+ 24813us) log.cc:844] Serialized 26 byte log entry
1023 02:04:56.443590 (+498923us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:56.444681 (+ 1091us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:56.444740 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:56.546095 (+101355us) raft_consensus.cc:1685] finished
1023 02:04:56.546116 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:56.641295 (+ 95179us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":211038,"spinlock_wait_cycles":4475136,"thread_start_us":365645,"threads_started":1,"wal-append.queue_time_us":465352}
W20251023 02:04:56.696853 21926 service_pool.cc:137] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53732 dropped due to backpressure: service queue is full with 50 items
W20251023 02:04:56.674957 21957 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 920) took 2528 ms (client timeout 1832 ms). Trace:
W20251023 02:04:56.698882 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:56.698820 21957 rpcz_store.cc:269] 1023 02:04:54.114871 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:54.140037 (+ 25166us) service_pool.cc:225] Handling call
1023 02:04:56.642624 (+2502587us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x7b54002dd7d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:56.642751 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1991008640}
I20251023 02:04:56.534179 23407 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.643572 23508 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:56.708525 23508 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.704838 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:56.643553 23532 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:56.642871 23555 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:56.711585 21957 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:56.700640 21926 service_pool.cc:138] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53732 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 941)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 943)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 947)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1231)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 946)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 954)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 944)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 950)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 956)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 940)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 951)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 957)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 945)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 953)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 955)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1229)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1211)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 933)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 98)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 99)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 100)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 938)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 948)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 949)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1235)
I20251023 02:04:56.616551 23500 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.646920 23440 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:56.674757 23501 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:56.811206 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.702818 23406 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:56.710549 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1198) took 2290 ms. Trace:
W20251023 02:04:56.853430 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 129534us
I20251023 02:04:56.648066 23540 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:56.785166 23374 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.891938 23352 consensus_queue.cc:1048] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:56.891422 23513 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:04:56.898538 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:56.840224 23477 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:56.923300 23477 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.793912 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.937283 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.938444 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.939723 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:56.940207 22032 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:56.941776 23530 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:56.942265 23530 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:56.792706 23508 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.911037 23592 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:56.912235 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:56.806198 23576 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:56.934522 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 61038.4us
I20251023 02:04:56.934965 23477 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.901643 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:56.847473 21953 rpcz_store.cc:276] 1023 02:04:54.420569 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:56.709598 (+2289029us) service_pool.cc:225] Handling call
1023 02:04:56.709946 (+ 348us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:56.710192 (+ 246us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:56.710228 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:56.710279 (+ 51us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:56.710315 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:56.710375 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:56.710491 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:56.936404 23513 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:56.936889 23606 log.cc:927] Time spent T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.291s user 0.001s sys 0.000s
I20251023 02:04:56.853672 23406 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.783176 23593 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:56.987294 23576 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:56.988102 23593 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:56.988900 23406 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.990126 23593 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:56.990417 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:56.990474 23576 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.984448 23592 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.968783 23586 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:56.842612 23532 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:57.004025 23500 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:56.969918 23407 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:56.810250 23555 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:57.002180 23578 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:56.938445 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:56.991317 21953 raft_consensus.cc:2393] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:57.006675 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:57.011783 22030 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 5
I20251023 02:04:57.005015 23592 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:57.008571 23574 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:57.039746 23586 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:56.985118 23530 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:57.048573 22076 leader_election.cc:336] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.749s (SENT)
I20251023 02:04:57.051425 23513 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:04:57.053628 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 61525.7us
I20251023 02:04:57.061528 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:57.063199 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:57.035980 23555 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:57.069726 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:57.065253 23407 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:57.082497 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:57.094172 22108 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:04:57.202905 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:57.204298 22108 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:57.126035 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:57.205435 22107 consensus_meta.cc:287] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.558s user 0.008s sys 0.002s
I20251023 02:04:57.225009 23540 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:04:57.257755 22076 leader_election.cc:336] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.832s (SENT)
I20251023 02:04:57.258543 22076 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:57.084894 23407 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:57.181605 23586 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:57.204690 23584 log.cc:927] Time spent T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.167s user 0.000s sys 0.001s
I20251023 02:04:57.285195 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:57.225989 23430 consensus_meta.cc:287] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.660s user 0.006s sys 0.003s
W20251023 02:04:57.262467 23309 consensus_meta.cc:287] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.781s user 0.007s sys 0.002s
W20251023 02:04:57.225849 22105 consensus_meta.cc:287] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.702s user 0.006s sys 0.002s
I20251023 02:04:57.226217 23513 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:57.236474 21953 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:57.472602 21927 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.018s sys 0.012s
W20251023 02:04:57.628643 21927 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.784s user 0.018s sys 0.012s
W20251023 02:04:57.630025 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:57.501765 21928 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.006s sys 0.029s
I20251023 02:04:57.509948 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 928) took 3128 ms. Trace:
W20251023 02:04:57.630945 21928 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.650s user 0.006s sys 0.030s
W20251023 02:04:57.519569 21846 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.016s sys 0.008s
W20251023 02:04:57.631445 21846 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.734s user 0.017s sys 0.008s
I20251023 02:04:57.360810 22030 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
W20251023 02:04:58.386372 22030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55770 (request call id 100) took 1905 ms (client timeout 2237 ms). Trace:
W20251023 02:04:58.386665 22030 rpcz_store.cc:269] 1023 02:04:56.481452 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:56.982738 (+501286us) service_pool.cc:225] Handling call
1023 02:04:58.386305 (+1403567us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.387526 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:04:58.388059 22030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1206) took 1575 ms (client timeout 1916 ms). Trace:
W20251023 02:04:58.388291 22030 rpcz_store.cc:269] 1023 02:04:56.812939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.387324 (+1574385us) service_pool.cc:225] Handling call
1023 02:04:58.388021 (+ 697us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.388967 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:57.353132 22029 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
W20251023 02:04:58.428040 22029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1197) took 1908 ms (client timeout 2100 ms). Trace:
W20251023 02:04:58.428431 22029 rpcz_store.cc:269] 1023 02:04:56.520065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:56.938120 (+418055us) service_pool.cc:225] Handling call
1023 02:04:58.427976 (+1489856us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51456}
W20251023 02:04:57.484421 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:58.431195 23593 consensus_queue.cc:1048] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:57.422425 22032 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:04:57.522286 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 935) took 3322 ms. Trace:
I20251023 02:04:58.514190 22028 rpcz_store.cc:276] 1023 02:04:54.199921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:54.200865 (+ 944us) service_pool.cc:225] Handling call
1023 02:04:54.209088 (+ 8223us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:54.374650 (+165562us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:57.522076 (+3147426us) spinlock_profiling.cc:243] Waited 67.4 ms on lock 0x7b54002ea754. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:57.522236 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":155001600}
I20251023 02:04:58.515846 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.516523 22028 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
W20251023 02:04:57.462083 22031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 938) took 2816 ms (client timeout 2201 ms). Trace:
I20251023 02:04:57.453680 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:57.429366 22037 raft_consensus.cc:1240] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20251023 02:04:58.522964 23643 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.523188 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 936) took 4092 ms. Trace:
I20251023 02:04:58.523489 23643 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.523622 22037 rpcz_store.cc:276] 1023 02:04:54.431069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:54.431896 (+ 827us) service_pool.cc:225] Handling call
1023 02:04:57.428792 (+2996896us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b58002d5a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:04:57.429001 (+ 209us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:58.522803 (+1093802us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.523130 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8937856}
I20251023 02:04:58.524919 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.525519 22037 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
W20251023 02:04:58.526001 22037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 968) took 1669 ms (client timeout 2091 ms). Trace:
I20251023 02:04:58.526103 23643 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:58.526356 22037 rpcz_store.cc:269] 1023 02:04:56.856478 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.524640 (+1668162us) service_pool.cc:225] Handling call
1023 02:04:58.525950 (+ 1310us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.527343 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:57.495277 22035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1187) took 2579 ms. Trace:
I20251023 02:04:57.537978 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 96) took 3103 ms. Trace:
W20251023 02:04:57.541155 22033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1190) took 2436 ms (client timeout 1767 ms). Trace:
I20251023 02:04:57.549196 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 97) took 2998 ms. Trace:
I20251023 02:04:57.550571 21955 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20251023 02:04:57.331393 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 476758us
I20251023 02:04:58.529034 21954 rpcz_store.cc:276] 1023 02:04:54.550317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.382880 (+832563us) service_pool.cc:225] Handling call
1023 02:04:55.398799 (+ 15919us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:04:55.399174 (+ 375us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:55.399207 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:04:55.475372 (+ 76165us) log.cc:844] Serialized 52 byte log entry
1023 02:04:56.437879 (+962507us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:04:56.595532 (+157653us) log.cc:844] Serialized 14 byte log entry
1023 02:04:56.696278 (+100746us) log.cc:844] Serialized 14 byte log entry
1023 02:04:56.781888 (+ 85610us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:56.803388 (+ 21500us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:57.548922 (+745534us) raft_consensus.cc:1685] finished
1023 02:04:57.548970 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:57.549131 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":290793,"prepare.queue_time_us":689557,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":88617,"raft.run_cpu_time_us":425,"raft.run_wall_time_us":50362,"thread_start_us":987053,"threads_started":2,"wal-append.queue_time_us":1555928}
I20251023 02:04:58.528702 21956 rpcz_store.cc:276] 1023 02:04:54.434922 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.381108 (+946186us) service_pool.cc:225] Handling call
1023 02:04:55.393938 (+ 12830us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:04:56.074760 (+680822us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:56.077679 (+ 2919us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:04:56.273114 (+195435us) log.cc:844] Serialized 574 byte log entry
1023 02:04:56.496888 (+223774us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:04:56.507726 (+ 10838us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:57.537437 (+1029711us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b540030f7d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:57.537504 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:57.537571 (+ 67us) raft_consensus.cc:1685] finished
1023 02:04:57.537615 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:57.537905 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1319,"raft.queue_time_us":72444,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":5530,"spinlock_wait_cycles":1180002048,"thread_start_us":151716,"threads_started":1,"wal-append.queue_time_us":270201,"wal-append.run_cpu_time_us":1662,"wal-append.run_wall_time_us":1001690}
W20251023 02:04:58.532039 21954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 939) took 2323 ms (client timeout 1749 ms). Trace:
W20251023 02:04:58.532320 21956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1231) took 2131 ms (client timeout 1558 ms). Trace:
W20251023 02:04:58.532433 21954 rpcz_store.cc:269] 1023 02:04:56.208464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.531869 (+2323405us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.531980 (+ 111us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20251023 02:04:58.532765 21956 rpcz_store.cc:269] 1023 02:04:56.400959 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.532151 (+2131192us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.532257 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:57.334710 22108 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251023 02:04:58.535136 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 122) took 1662 ms. Trace:
I20251023 02:04:58.535466 22108 rpcz_store.cc:276] 1023 02:04:56.872735 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:57.327721 (+454986us) service_pool.cc:225] Handling call
1023 02:04:57.334298 (+ 6577us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.534886 (+1200588us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.535085 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.536715 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:57.542944 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:57.414506 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 116) took 1084 ms. Trace:
W20251023 02:04:58.539024 22111 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56894 (request call id 113) took 2957 ms (client timeout 1800 ms). Trace:
I20251023 02:04:57.327163 22105 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
W20251023 02:04:58.533478 21954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1224) took 2207 ms (client timeout 1684 ms). Trace:
I20251023 02:04:57.309842 22587 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:58.543768 23464 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251023 02:04:58.543915 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 1.25717e+06us
I20251023 02:04:58.544212 23464 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:58.544661 23464 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:58.544729 22587 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:58.545634 23397 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:57.368337 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.548097 22615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 186) took 1185 ms. Trace:
I20251023 02:04:58.548482 22615 rpcz_store.cc:276] 1023 02:04:57.362927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:57.367903 (+ 4976us) service_pool.cc:225] Handling call
1023 02:04:58.548026 (+1180123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:57.368582 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.550631 22616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 188) took 1182 ms. Trace:
I20251023 02:04:58.550994 22616 rpcz_store.cc:276] 1023 02:04:57.367910 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:57.368174 (+ 264us) service_pool.cc:225] Handling call
1023 02:04:58.550566 (+1182392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:57.367355 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:57.366375 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:57.425050 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.554414 22620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 184) took 1192 ms. Trace:
I20251023 02:04:57.354955 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:57.361989 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:58.556624 22624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 182) took 1195 ms (client timeout 1508 ms). Trace:
W20251023 02:04:58.556979 22624 rpcz_store.cc:269] 1023 02:04:57.361639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:57.361751 (+ 112us) service_pool.cc:225] Handling call
1023 02:04:58.556569 (+1194818us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:57.369393 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.559002 22617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 187) took 1195 ms. Trace:
I20251023 02:04:58.559367 22617 rpcz_store.cc:276] 1023 02:04:57.363284 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:57.368937 (+ 5653us) service_pool.cc:225] Handling call
1023 02:04:58.558947 (+1190010us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:57.417894 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.562297 22619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 181) took 1203 ms. Trace:
I20251023 02:04:58.562409 23456 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251023 02:04:58.562665 22619 rpcz_store.cc:276] 1023 02:04:57.359000 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:57.362418 (+ 3418us) service_pool.cc:225] Handling call
1023 02:04:58.562241 (+1199823us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:57.403445 22621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 177) took 2372 ms (client timeout 2056 ms). Trace:
W20251023 02:04:58.564834 22621 rpcz_store.cc:269] 1023 02:04:55.030996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.031161 (+ 165us) service_pool.cc:225] Handling call
1023 02:04:57.403125 (+2371964us) spinlock_profiling.cc:243] Waited 62.2 ms on lock 0x7b54005ce9d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:57.403326 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":143021440}
I20251023 02:04:57.595551 23216 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:57.326661 23309 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:57.351861 23407 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:57.309474 23430 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.571731 22109 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:58.572098 23592 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:04:58.573038 23430 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:58.573192 23309 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:57.547567 23473 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:57.368937 23485 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:58.583915 23485 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.585462 23485 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:57.330358 23513 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:57.264114 23521 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:58.587248 23521 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.587816 23513 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:58.589056 23521 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:57.227073 23532 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:57.347117 23535 log.cc:927] Time spent T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 2.618s user 0.017s sys 0.007s
I20251023 02:04:57.454483 23543 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:57.562757 23542 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:58.592579 23543 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:57.558140 23546 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 2.621s user 0.020s sys 0.030s
W20251023 02:04:57.329308 23558 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.002s sys 0.018s
I20251023 02:04:58.596561 23543 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:58.596832 23558 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.872s user 0.004s sys 0.019s
W20251023 02:04:57.402467 23616 log.cc:927] Time spent T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20251023 02:04:57.625530 21926 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.012s sys 0.015s
W20251023 02:04:57.631279 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 778283us
W20251023 02:04:58.598551 21926 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.753s user 0.013s sys 0.015s
W20251023 02:04:58.600677 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:57.384156 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:58.602422 21926 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53734 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 954)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 944)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 950)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 956)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 940)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 951)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 957)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 945)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 953)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 955)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 962)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 961)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 963)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1242)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1211)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 933)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 98)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 99)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 100)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 938)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 948)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 949)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 958)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1239)
I20251023 02:04:58.601939 23473 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.609478 23473 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.610873 23554 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.611370 23554 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.612177 23407 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.612326 23473 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:58.612872 23473 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.612752 23407 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:58.613191 23554 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:57.631731 21846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 778504us
I20251023 02:04:58.613348 23473 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:57.631014 21961 rpcz_store.cc:276] 1023 02:04:54.381211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.145842 (+764631us) service_pool.cc:225] Handling call
1023 02:04:55.191072 (+ 45230us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:55.200365 (+ 9293us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:57.509341 (+2308976us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b540017dc50. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:57.509501 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":102936448}
I20251023 02:04:57.639927 21953 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
W20251023 02:04:58.614681 21953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1227) took 2269 ms (client timeout 1585 ms). Trace:
I20251023 02:04:58.614934 23407 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:58.615206 23473 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:58.615077 21953 rpcz_store.cc:269] 1023 02:04:56.345822 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:57.067924 (+722102us) service_pool.cc:225] Handling call
1023 02:04:58.614619 (+1546695us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:57.723196 21925 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53744 dropped due to backpressure: service queue is full with 50 items
I20251023 02:04:57.735231 23297 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:58.091270 23630 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:04:58.100248 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:58.616878 23630 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.226541 23406 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.618170 23406 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.607311 21926 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53734 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 954)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 944)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 950)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 956)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 940)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 951)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 957)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 945)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 953)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 955)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 962)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 961)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 963)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1242)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1211)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 933)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 98)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 99)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 100)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 938)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 948)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 949)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 958)
I20251023 02:04:58.334702 23467 raft_consensus.cc:3055] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
W20251023 02:04:58.349749 21962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 941) took 2140 ms (client timeout 1572 ms). Trace:
I20251023 02:04:58.352638 23385 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:58.621685 21983 debug-util.cc:398] Leaking SignalData structure 0x7b08000e6780 after lost signal to thread 20807
W20251023 02:04:58.621920 21962 rpcz_store.cc:269] 1023 02:04:56.209048 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.349493 (+2140445us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.349657 (+ 164us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.622109 23385 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:58.622727 21983 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce600 after lost signal to thread 21894
W20251023 02:04:58.622964 21962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 954) took 2097 ms (client timeout 1570 ms). Trace:
W20251023 02:04:58.623121 21983 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce720 after lost signal to thread 21908
W20251023 02:04:58.623358 21962 rpcz_store.cc:269] 1023 02:04:56.525840 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.622788 (+2096948us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.622907 (+ 119us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.624099 23385 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:04:58.624138 21962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 944) took 2358 ms (client timeout 1832 ms). Trace:
I20251023 02:04:58.354051 23547 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
W20251023 02:04:58.624648 21962 rpcz_store.cc:269] 1023 02:04:56.266099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.623985 (+2357886us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.624087 (+ 102us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.624859 23547 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
W20251023 02:04:58.625602 21962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 956) took 2016 ms (client timeout 1532 ms). Trace:
W20251023 02:04:58.625958 21962 rpcz_store.cc:269] 1023 02:04:56.609373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.625421 (+2016048us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.625530 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20251023 02:04:58.626294 21961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 950) took 2220 ms (client timeout 1724 ms). Trace:
W20251023 02:04:58.626778 21962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 940) took 2418 ms (client timeout 1971 ms). Trace:
I20251023 02:04:58.359000 23629 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:54.728175 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:04:58.616828 23297 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:58.373672 21957 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 943) took 2122 ms (client timeout 1577 ms). Trace:
I20251023 02:04:58.380386 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 94) took 3977 ms. Trace:
W20251023 02:04:58.389460 22030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1196) took 1869 ms (client timeout 2235 ms). Trace:
I20251023 02:04:58.396045 23404 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:58.396425 23520 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:58.405366 23352 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:58.633854 23352 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:58.629473 23297 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:58.635959 23352 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:58.633814 23520 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.417776 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:58.419160 22103 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1248) took 2307 ms (client timeout 2193 ms). Trace:
I20251023 02:04:58.636631 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:58.637020 22103 rpcz_store.cc:269] 1023 02:04:56.111608 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.418930 (+2307322us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.419063 (+ 133us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.637766 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:58.638038 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.638339 22034 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
I20251023 02:04:58.638720 22103 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:58.638871 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.638144 23520 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:58.624296 21983 debug-util.cc:398] Leaking SignalData structure 0x7b0800321c20 after lost signal to thread 21986
I20251023 02:04:58.438037 23522 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.438306 23005 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
W20251023 02:04:58.629736 21957 rpcz_store.cc:269] 1023 02:04:56.251686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.373466 (+2121780us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.373572 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.640232 23005 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:58.640798 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:58.641148 21957 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 957) took 2012 ms (client timeout 1618 ms). Trace:
I20251023 02:04:58.433804 22590 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:04:58.630213 22030 rpcz_store.cc:269] 1023 02:04:56.519770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.388784 (+1869014us) service_pool.cc:225] Handling call
1023 02:04:58.389428 (+ 644us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:58.641985 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 208295us
I20251023 02:04:58.642623 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.642872 23005 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:58.429477 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.443630 23593 consensus_queue.cc:1048] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20251023 02:04:58.460626 22036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 946) took 2905 ms (client timeout 2235 ms). Trace:
I20251023 02:04:58.461396 23637 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:58.467319 23214 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:58.469031 21958 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 914) took 4355 ms (client timeout 1749 ms). Trace:
W20251023 02:04:58.643738 22029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 963) took 1996 ms (client timeout 2152 ms). Trace:
I20251023 02:04:58.457471 23493 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:58.481432 23639 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.482261 23640 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:04:58.644385 22029 rpcz_store.cc:269] 1023 02:04:56.647420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.429209 (+1781789us) service_pool.cc:225] Handling call
1023 02:04:58.643680 (+214471us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.645092 23640 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.645542 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.645963 23214 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:58.601351 21928 connection.cc:232] RPC call timeout handler was delayed by -1.3014758580000034s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
I20251023 02:04:58.647341 23640 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:58.496263 21859 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56872 (request call id 212) took 7425 ms. Trace:
W20251023 02:04:58.517041 22028 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1208) took 1531 ms (client timeout 1910 ms). Trace:
I20251023 02:04:58.647944 23214 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:58.500283 22032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 965) took 1852 ms (client timeout 1994 ms). Trace:
I20251023 02:04:58.648296 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.519743 21856 catalog_manager.cc:5649] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:58.459609 23635 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:58.640112 23522 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.649458 23635 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.629920 21959 rpcz_store.cc:276] 1023 02:04:54.402685 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.374872 (+3972187us) service_pool.cc:225] Handling call
1023 02:04:58.375236 (+ 364us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:58.375556 (+ 320us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:58.375586 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:58.376028 (+ 442us) log.cc:844] Serialized 26 byte log entry
1023 02:04:58.377013 (+ 985us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:58.377052 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.377159 (+ 107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:58.380160 (+ 3001us) raft_consensus.cc:1685] finished
1023 02:04:58.380181 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:58.380328 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":979,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":635,"threads_started":1,"wal-append.queue_time_us":900}
I20251023 02:04:58.650161 21856 catalog_manager.cc:5649] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
W20251023 02:04:58.651103 21959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 945) took 2380 ms (client timeout 1982 ms). Trace:
W20251023 02:04:58.651432 21959 rpcz_store.cc:269] 1023 02:04:56.270274 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.650948 (+2380674us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.651052 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.651641 23635 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:58.482025 22589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.759s (SENT) [suppressed 26 similar messages]
I20251023 02:04:58.529606 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 93) took 4134 ms. Trace:
W20251023 02:04:58.528970 22033 rpcz_store.cc:269] 1023 02:04:55.104286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.104417 (+ 131us) service_pool.cc:225] Handling call
1023 02:04:57.540908 (+2436491us) spinlock_profiling.cc:243] Waited 957 ms on lock 0x7b54002a80d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:57.541081 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2201507584}
I20251023 02:04:58.512604 23641 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:04:58.520157 22031 rpcz_store.cc:269] 1023 02:04:54.645977 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:54.649617 (+ 3640us) service_pool.cc:225] Handling call
1023 02:04:57.461876 (+2812259us) spinlock_profiling.cc:243] Waited 42.5 ms on lock 0x7b54003a9154. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:57.461990 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":97763200}
I20251023 02:04:57.372764 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 118) took 1035 ms. Trace:
W20251023 02:04:58.540666 22105 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56894 (request call id 115) took 2210 ms (client timeout 1837 ms). Trace:
I20251023 02:04:57.484367 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1238) took 1938 ms. Trace:
I20251023 02:04:58.539208 22107 rpcz_store.cc:276] 1023 02:04:56.330537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:56.509626 (+179089us) service_pool.cc:225] Handling call
1023 02:04:56.529731 (+ 20105us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:56.529988 (+ 257us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:04:57.227862 (+697874us) log.cc:844] Serialized 14 byte log entry
1023 02:04:57.413968 (+186106us) log.cc:844] Serialized 14 byte log entry
1023 02:04:57.414176 (+ 208us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:57.414229 (+ 53us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:57.414268 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:57.414343 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:57.414456 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":77121,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":32,"raft.queue_time_us":375,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":490,"thread_start_us":169627,"threads_started":2,"wal-append.queue_time_us":194178,"wal-append.run_cpu_time_us":1473,"wal-append.run_wall_time_us":1001507}
I20251023 02:04:58.545984 23397 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:04:58.546408 21954 rpcz_store.cc:269] 1023 02:04:56.326391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.533289 (+2206898us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.533420 (+ 131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20251023 02:04:58.553663 22618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 185) took 1191 ms (client timeout 1584 ms). Trace:
I20251023 02:04:58.554220 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.554808 22620 rpcz_store.cc:276] 1023 02:04:57.362239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:57.365915 (+ 3676us) service_pool.cc:225] Handling call
1023 02:04:58.554360 (+1188445us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.555866 22623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34820 (request call id 230) took 1201 ms. Trace:
W20251023 02:04:58.567142 22077 connection.cc:232] RPC call timeout handler was delayed by -1.5331166060001351s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2235s for a response
I20251023 02:04:58.567905 23464 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:58.574124 22109 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56894 (request call id 119) took 2051 ms (client timeout 2237 ms). Trace:
I20251023 02:04:58.574687 22108 raft_consensus.cc:2393] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:58.583078 23585 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.593099 23542 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.595014 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 89) took 4539 ms. Trace:
I20251023 02:04:58.618981 23630 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:58.320294 23569 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:58.620491 23406 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:58.626694 21961 rpcz_store.cc:269] 1023 02:04:56.406352 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.626139 (+2219787us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.626238 (+ 99us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.354812 23541 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:58.621837 23467 raft_consensus.cc:740] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
W20251023 02:04:58.627314 21962 rpcz_store.cc:269] 1023 02:04:56.208754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.626621 (+2417867us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.626727 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.626891 23547 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:58.627506 23629 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:58.652282 21959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 953) took 2161 ms (client timeout 1772 ms). Trace:
W20251023 02:04:58.686686 21962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1232) took 2268 ms (client timeout 2052 ms). Trace:
I20251023 02:04:58.664525 23522 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:58.687182 21962 rpcz_store.cc:269] 1023 02:04:56.418375 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.686528 (+2268153us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.686604 (+ 76us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.687883 23522 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:58.686897 23541 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:58.652631 21953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 951) took 2209 ms (client timeout 1775 ms). Trace:
I20251023 02:04:58.417034 22102 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20251023 02:04:58.639724 21983 debug-util.cc:398] Leaking SignalData structure 0x7b0800322f40 after lost signal to thread 21987
I20251023 02:04:58.642081 23456 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:04:58.690677 23456 raft_consensus.cc:697] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:58.643337 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1203) took 2108 ms. Trace:
W20251023 02:04:58.691314 21983 debug-util.cc:398] Leaking SignalData structure 0x7b080028a4c0 after lost signal to thread 22061
I20251023 02:04:58.645049 23639 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.691737 23456 consensus_queue.cc:237] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:58.692569 21983 debug-util.cc:398] Leaking SignalData structure 0x7b08002ea500 after lost signal to thread 22135
W20251023 02:04:58.692889 21983 debug-util.cc:398] Leaking SignalData structure 0x7b080026f400 after lost signal to thread 22136
I20251023 02:04:58.693275 23639 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:58.646462 22029 raft_consensus.cc:2410] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20251023 02:04:58.688382 23585 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.694391 23541 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:58.694077 23467 consensus_queue.cc:260] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:58.643754 22036 rpcz_store.cc:269] 1023 02:04:55.555288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:56.073851 (+518563us) service_pool.cc:225] Handling call
1023 02:04:58.460377 (+2386526us) spinlock_profiling.cc:243] Waited 67.3 ms on lock 0x7b540001e0d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:58.460533 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":154891008}
W20251023 02:04:58.533990 21956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 946) took 2255 ms (client timeout 1815 ms). Trace:
I20251023 02:04:58.695086 23467 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:58.694718 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1200) took 2173 ms. Trace:
I20251023 02:04:58.695399 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.695672 22029 rpcz_store.cc:276] 1023 02:04:56.520923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.645292 (+2124369us) service_pool.cc:225] Handling call
1023 02:04:58.694658 (+ 49366us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.696166 22036 raft_consensus.cc:2393] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:58.696412 23585 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:58.696635 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.696982 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.697279 22029 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:58.522258 23642 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:58.528569 22035 rpcz_store.cc:276] 1023 02:04:54.915489 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:54.915612 (+ 123us) service_pool.cc:225] Handling call
1023 02:04:54.924697 (+ 9085us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:55.158012 (+233315us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:57.494965 (+2336953us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b540027c254. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:57.495169 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":535290240}
W20251023 02:04:58.712442 22000 leader_election.cc:336] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899): Timed out: RequestConsensusVote RPC to 127.20.81.3:33899 timed out after 1.703s (SENT)
I20251023 02:04:58.534384 23643 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:04:58.539517 22111 rpcz_store.cc:269] 1023 02:04:55.581527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.793222 (+211695us) service_pool.cc:225] Handling call
1023 02:04:57.542773 (+1749551us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:04:58.538918 (+996145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5422464}
I20251023 02:04:58.653239 21856 catalog_manager.cc:5649] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:04:58.689539 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.701094 21960 rpcz_store.cc:276] 1023 02:04:54.055612 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:54.075069 (+ 19457us) service_pool.cc:225] Handling call
1023 02:04:54.667590 (+592521us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b5800002a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:04:54.667762 (+ 172us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:54.668124 (+ 362us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:04:54.668151 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:54.668579 (+ 428us) log.cc:844] Serialized 26 byte log entry
1023 02:04:54.731027 (+ 62448us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:54.731096 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:54.731173 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:57.557146 (+2825973us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:04:58.594771 (+1037625us) raft_consensus.cc:1685] finished
1023 02:04:58.594795 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:58.594926 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2425,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":10386,"spinlock_wait_cycles":15831552,"thread_start_us":61668,"threads_started":1,"wal-append.queue_time_us":268407}
I20251023 02:04:58.690723 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 114) took 2361 ms. Trace:
I20251023 02:04:58.726087 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:58.726234 22102 rpcz_store.cc:276] 1023 02:04:56.329811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:56.331443 (+ 1632us) service_pool.cc:225] Handling call
1023 02:04:58.416513 (+2085070us) spinlock_profiling.cc:243] Waited 49.3 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:04:58.416756 (+ 243us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:58.690362 (+273606us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.690644 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113501568}
I20251023 02:04:58.688393 23522 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:58.727340 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.727747 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:58.654543 22034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1194) took 3240 ms (client timeout 2100 ms). Trace:
W20251023 02:04:58.697811 21983 debug-util.cc:398] Leaking SignalData structure 0x7b08003324c0 after lost signal to thread 22648
W20251023 02:04:58.716445 21954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 955) took 2169 ms (client timeout 2091 ms). Trace:
W20251023 02:04:58.728443 21983 debug-util.cc:398] Leaking SignalData structure 0x7b08002f7fc0 after lost signal to thread 22649
I20251023 02:04:58.699386 23542 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:58.699712 22109 rpcz_store.cc:269] 1023 02:04:56.523011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:56.535078 (+ 12067us) service_pool.cc:225] Handling call
1023 02:04:58.573948 (+2038870us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54001ab8d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:58.574061 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4428533632}
I20251023 02:04:58.697875 22616 raft_consensus.cc:1240] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:58.729386 22102 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:58.729849 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.730446 22109 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:58.730490 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.700430 23569 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:58.695040 21956 rpcz_store.cc:269] 1023 02:04:56.278786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.533838 (+2255052us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.533942 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.731276 22616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 231) took 1352 ms. Trace:
W20251023 02:04:58.688210 21962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1237) took 2045 ms (client timeout 1875 ms). Trace:
I20251023 02:04:58.731950 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:58.732216 21962 rpcz_store.cc:269] 1023 02:04:56.642895 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.688057 (+2045162us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.688157 (+ 100us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.732517 21956 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:58.732728 23569 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:58.733131 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.734028 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.701524 22077 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:58.735244 22615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 190) took 1318 ms. Trace:
I20251023 02:04:58.671351 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.735580 22077 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:58.653371 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.697952 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.736495 22031 raft_consensus.cc:2410] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20251023 02:04:58.703862 22618 rpcz_store.cc:269] 1023 02:04:57.362593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:57.366981 (+ 4388us) service_pool.cc:225] Handling call
1023 02:04:58.553580 (+1186599us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.736954 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1199) took 2216 ms. Trace:
I20251023 02:04:58.701455 23543 consensus_queue.cc:1059] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:04:58.737440 23643 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.737990 23543 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:58.731343 22102 raft_consensus.cc:2393] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:58.738418 23543 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:58.737289 22031 rpcz_store.cc:276] 1023 02:04:56.520652 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.653146 (+2132494us) service_pool.cc:225] Handling call
1023 02:04:58.736900 (+ 83754us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.739212 23630 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:58.739459 23643 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:58.739652 23630 raft_consensus.cc:697] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:58.739996 23643 raft_consensus.cc:3055] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:04:58.740392 23643 raft_consensus.cc:740] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:58.740574 23630 consensus_queue.cc:237] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.702092 23397 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:58.647876 21859 rpcz_store.cc:276] 1023 02:04:51.071179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.071300 (+ 121us) service_pool.cc:225] Handling call
1023 02:04:58.496167 (+7424867us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:04:53.779899 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:04:53.780586 (+ 687us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:04:53.780641 (+ 55us) write_op.cc:435] Acquired schema lock
1023 02:04:53.780657 (+ 16us) tablet.cc:660] Decoding operations
1023 02:04:53.844561 (+ 63904us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:04:53.866224 (+ 21663us) write_op.cc:641] Partition lock acquired for write op
1023 02:04:53.869393 (+ 3169us) tablet.cc:683] Acquiring locks for 6 operations
1023 02:04:53.912388 (+ 42995us) tablet.cc:699] Row locks acquired
1023 02:04:53.912410 (+ 22us) write_op.cc:260] PREPARE: finished
1023 02:04:53.912558 (+ 148us) write_op.cc:270] Start()
1023 02:04:53.912712 (+ 154us) write_op.cc:276] Timestamp: P: 1761185093912499 usec, L: 0
1023 02:04:53.912731 (+ 19us) op_driver.cc:348] REPLICATION: starting
1023 02:04:53.913320 (+ 589us) log.cc:844] Serialized 2435 byte log entry
1023 02:04:54.213703 (+300383us) op_driver.cc:464] REPLICATION: finished
1023 02:04:54.446521 (+232818us) write_op.cc:301] APPLY: starting
1023 02:04:54.455204 (+ 8683us) tablet.cc:1365] starting BulkCheckPresence
1023 02:04:54.493316 (+ 38112us) tablet.cc:1368] finished BulkCheckPresence
1023 02:04:54.553272 (+ 59956us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:04:54.722410 (+169138us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:04:54.722589 (+ 179us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
1023 02:04:54.722643 (+ 54us) write_op.cc:312] APPLY: finished
1023 02:04:54.722900 (+ 257us) log.cc:844] Serialized 52 byte log entry
1023 02:04:54.723065 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:54.723173 (+ 108us) write_op.cc:454] Released schema lock
1023 02:04:54.723222 (+ 49us) write_op.cc:341] FINISH: Updating metrics
1023 02:04:54.723594 (+ 372us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:04:54.723626 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":920657,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":227891,"apply.run_cpu_time_us":3788,"apply.run_wall_time_us":277054,"num_ops":6,"prepare.queue_time_us":199494,"prepare.run_cpu_time_us":4167,"prepare.run_wall_time_us":160236,"replication_time_us":279681,"thread_start_us":312760,"threads_started":2}]]}
I20251023 02:04:58.735258 23522 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:58.731307 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.739486 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:58.743502 22109 raft_consensus.cc:2368] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:04:58.743621 22031 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:58.744303 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.744872 22109 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:58.652843 23641 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.640080 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:04:58.647965 22028 rpcz_store.cc:269] 1023 02:04:56.985991 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.515511 (+1529520us) service_pool.cc:225] Handling call
1023 02:04:58.516991 (+ 1480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.636468 23352 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:58.688357 23629 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:58.747300 23352 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.747627 23642 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.731716 22616 rpcz_store.cc:276] 1023 02:04:57.378712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.551702 (+1172990us) service_pool.cc:225] Handling call
1023 02:04:58.552141 (+ 439us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.729604 (+177463us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.731029 (+ 1425us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54000744d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:58.731219 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":334703360}
I20251023 02:04:58.749202 23352 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:58.629742 22037 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:58.701170 21926 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53734 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 955)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 962)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 961)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1253)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1251)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1248)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 963)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1250)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1211)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 933)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 98)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 99)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 100)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 938)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 948)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 949)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 958)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1246)
I20251023 02:04:58.749949 23642 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:04:58.750366 22037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 957) took 2347 ms (client timeout 2587 ms). Trace:
W20251023 02:04:58.750733 22037 rpcz_store.cc:269] 1023 02:04:56.403043 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.527080 (+2124037us) service_pool.cc:225] Handling call
1023 02:04:58.750217 (+223137us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540027b854. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:58.750310 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":234190720}
I20251023 02:04:58.735616 22615 rpcz_store.cc:276] 1023 02:04:57.416666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.549417 (+1132751us) service_pool.cc:225] Handling call
1023 02:04:58.735183 (+185766us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.752599 22002 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:58.752815 22028 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:58.753454 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 101) took 2260 ms. Trace:
W20251023 02:04:58.644215 21958 rpcz_store.cc:269] 1023 02:04:54.113346 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:54.114280 (+ 934us) service_pool.cc:225] Handling call
1023 02:04:58.468835 (+4354555us) spinlock_profiling.cc:243] Waited 897 ms on lock 0x7b54000c3d54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:58.468946 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2063262464}
I20251023 02:04:58.652449 21955 rpcz_store.cc:276] 1023 02:04:54.394811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.149432 (+754621us) service_pool.cc:225] Handling call
1023 02:04:57.550050 (+2400618us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b5800002a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:04:57.550295 (+ 245us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:58.529299 (+979004us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.529524 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44883328}
W20251023 02:04:58.652595 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 170630us
I20251023 02:04:58.702152 23639 consensus_queue.cc:1059] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20251023 02:04:58.648581 23637 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.686271 22623 rpcz_store.cc:276] 1023 02:04:57.354555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:57.354693 (+ 138us) service_pool.cc:225] Handling call
1023 02:04:58.555805 (+1201112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.688200 23464 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:58.639657 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.795814 23585 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:58.797612 23585 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:58.798094 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.798794 22103 raft_consensus.cc:2393] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:58.799539 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.800056 23585 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:58.800597 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.801169 22103 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
W20251023 02:04:58.648630 22032 rpcz_store.cc:269] 1023 02:04:56.648022 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:56.939255 (+291233us) service_pool.cc:225] Handling call
1023 02:04:58.500203 (+1560948us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.802310 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.803973 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 102) took 2287 ms. Trace:
I20251023 02:04:58.804283 22032 rpcz_store.cc:276] 1023 02:04:56.516201 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.803222 (+2287021us) service_pool.cc:225] Handling call
1023 02:04:58.803496 (+ 274us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.803658 (+ 162us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:58.803684 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:58.803728 (+ 44us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:04:58.803759 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.803817 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:58.803923 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:58.641517 21957 rpcz_store.cc:269] 1023 02:04:56.628629 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.640989 (+2012360us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.641094 (+ 105us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20251023 02:04:58.686563 21959 rpcz_store.cc:269] 1023 02:04:56.490885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.652127 (+2161242us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.652233 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.806589 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.807337 21959 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:58.808238 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.808864 21959 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
W20251023 02:04:58.686643 21961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1236) took 2136 ms (client timeout 1820 ms). Trace:
W20251023 02:04:58.810498 21961 rpcz_store.cc:269] 1023 02:04:56.550408 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.686467 (+2136059us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.686581 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.811566 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.812812 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.722188 22000 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:58.814296 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 101891us
I20251023 02:04:58.816800 22000 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:58.715209 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.703073 22110 rpcz_store.cc:276] 1023 02:04:55.545563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.558599 (+ 13036us) service_pool.cc:225] Handling call
1023 02:04:57.483646 (+1925047us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:04:57.483863 (+ 217us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:57.484027 (+ 164us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:57.484042 (+ 15us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:57.484076 (+ 34us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:57.484106 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:57.484168 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:57.484306 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2895744}
I20251023 02:04:58.727969 22035 raft_consensus.cc:2368] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4.
W20251023 02:04:58.728339 22034 rpcz_store.cc:269] 1023 02:04:55.414441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:55.414582 (+ 141us) service_pool.cc:225] Handling call
1023 02:04:57.453470 (+2038888us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b58002d5a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:04:58.654397 (+1200927us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b540027bd54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:58.654488 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":310120960}
I20251023 02:04:58.737636 23569 consensus_queue.cc:1059] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:04:58.738808 23543 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:58.691469 22030 rpcz_store.cc:276] 1023 02:04:56.534983 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.642373 (+2107390us) service_pool.cc:225] Handling call
1023 02:04:58.643284 (+ 911us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.702762 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:58.828172 22036 raft_consensus.cc:2368] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5.
I20251023 02:04:58.746798 23641 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:58.652989 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.688524 22106 rpcz_store.cc:276] 1023 02:04:56.345512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:56.648721 (+303209us) service_pool.cc:225] Handling call
1023 02:04:56.701333 (+ 52612us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:56.853370 (+152037us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:56.853399 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:56.873304 (+ 19905us) log.cc:844] Serialized 26 byte log entry
1023 02:04:57.310357 (+437053us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:57.310398 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:57.310466 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:57.372524 (+ 62058us) raft_consensus.cc:1685] finished
1023 02:04:57.372552 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:57.372697 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":150001,"raft.queue_time_us":150213,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":19177,"thread_start_us":249771,"threads_started":1,"wal-append.queue_time_us":281828,"wal-append.run_cpu_time_us":1921,"wal-append.run_wall_time_us":1026487}
I20251023 02:04:58.616251 21925 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53744 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 951)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 957)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 945)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 953)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 955)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 962)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 961)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 963)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1242)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1211)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 933)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 98)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 99)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 100)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 938)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 948)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 949)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 958)
W20251023 02:04:58.759063 21926 service_pool.cc:137] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53732 dropped due to backpressure: service queue is full with 50 items [suppressed 3 similar messages]
I20251023 02:04:58.751708 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.755745 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.837090 22109 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:58.769071 23541 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:58.770839 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.771780 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.772714 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:58.774546 21956 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:58.753818 22028 rpcz_store.cc:276] 1023 02:04:56.493278 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.752162 (+2258884us) service_pool.cc:225] Handling call
1023 02:04:58.752465 (+ 303us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.753219 (+ 754us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.753404 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:58.686892 22105 rpcz_store.cc:269] 1023 02:04:56.330202 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:56.331286 (+ 1084us) service_pool.cc:225] Handling call
1023 02:04:58.540598 (+2209312us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:58.691344 21953 rpcz_store.cc:269] 1023 02:04:56.443097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.652451 (+2209354us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.652571 (+ 120us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.780474 23502 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.740619 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.781996 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:58.782573 23406 consensus_queue.cc:1059] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251023 02:04:58.873164 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.783548 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:04:58.876320 21958 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1229) took 2478 ms (client timeout 3302 ms). Trace:
W20251023 02:04:58.876590 21958 rpcz_store.cc:269] 1023 02:04:56.397804 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.783336 (+2385532us) service_pool.cc:225] Handling call
1023 02:04:58.876274 (+ 92938us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.877312 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.877746 21958 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:04:58.878307 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.726079 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.794278 21928 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:58.880568 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:58.880887 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 86691.1us
I20251023 02:04:58.881198 21953 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:58.881443 23637 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:58.881856 21928 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:58.882151 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.883278 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.789486 23397 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:58.883517 21928 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:58.883654 23404 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:04:58.884334 23404 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:58.884541 21928 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:58.884732 23297 consensus_queue.cc:1059] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:04:58.819583 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.795085 22075 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:58.885565 22622 raft_consensus.cc:2393] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 2: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:58.885831 23491 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:58.886076 23642 raft_consensus.cc:3055] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:58.886219 23491 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:58.886533 23642 raft_consensus.cc:740] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:04:58.886703 23491 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:58.886888 23585 raft_consensus.cc:3055] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:58.887710 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.887388 23642 consensus_queue.cc:260] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:58.888100 22621 raft_consensus.cc:1240] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:04:58.888255 22111 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:58.888746 23642 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:58.889143 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.889662 22111 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:58.889746 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.824316 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.802248 22031 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:04:58.834597 22036 raft_consensus.cc:1275] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:58.837173 22037 raft_consensus.cc:2410] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:04:58.836251 22033 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:58.898936 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 962) took 2252 ms. Trace:
I20251023 02:04:58.838080 22108 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:58.899384 22036 rpcz_store.cc:276] 1023 02:04:56.647064 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.828665 (+2181601us) service_pool.cc:225] Handling call
1023 02:04:58.834164 (+ 5499us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.898709 (+ 64545us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.898890 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.793215 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:58.843014 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:58.901633 22107 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:58.823604 23493 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.845702 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.846045 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:58.903628 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 107) took 1839 ms. Trace:
I20251023 02:04:58.903908 22110 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:58.840615 22035 raft_consensus.cc:2955] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:58.826480 23473 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.825416 23572 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:58.836630 23547 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:58.907245 23005 raft_consensus.cc:3055] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:04:58.907334 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.907673 23005 raft_consensus.cc:740] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:58.907972 22111 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:58.879601 22102 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:04:58.827574 23521 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:04:58.729063 21954 rpcz_store.cc:269] 1023 02:04:56.546694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.716248 (+2169554us) service_pool.cc:212] Skipping call since client already timed out
1023 02:04:58.716381 (+ 133us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:04:58.872619 22030 raft_consensus.cc:1275] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:58.843838 22104 raft_consensus.cc:2376] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5. Re-sending same reply.
I20251023 02:04:58.839926 23541 raft_consensus.cc:697] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:58.915048 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.874600 23502 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.916261 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.915580 23541 consensus_queue.cc:237] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:04:58.917094 22104 raft_consensus.cc:2410] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251023 02:04:58.907001 23473 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.918430 23502 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:58.918936 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:58.919768 21962 raft_consensus.cc:2410] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:04:58.920753 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 962) took 1198 ms. Trace:
I20251023 02:04:58.920994 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.921048 21962 rpcz_store.cc:276] 1023 02:04:57.722335 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.732878 (+1010543us) service_pool.cc:225] Handling call
1023 02:04:58.920594 (+187716us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b5400107554. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:58.920704 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":406646144}
I20251023 02:04:58.922211 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.902262 21953 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:58.905959 23547 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:58.840241 23532 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:58.913673 22617 raft_consensus.cc:2393] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:04:58.923195 23547 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:58.923292 23532 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:58.924068 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 108) took 1859 ms. Trace:
I20251023 02:04:58.924448 22031 rpcz_store.cc:276] 1023 02:04:57.064618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.903030 (+1838412us) service_pool.cc:225] Handling call
1023 02:04:58.911370 (+ 8340us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.911570 (+ 200us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:58.911596 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:58.911657 (+ 61us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:04:58.913861 (+ 2204us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.923903 (+ 10042us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:58.924006 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.925374 23532 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:58.879616 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.926254 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 109) took 1860 ms. Trace:
I20251023 02:04:58.879982 21958 raft_consensus.cc:2393] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:58.926615 22031 rpcz_store.cc:276] 1023 02:04:57.065834 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.925439 (+1859605us) service_pool.cc:225] Handling call
1023 02:04:58.925773 (+ 334us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.925941 (+ 168us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:58.925973 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:58.926015 (+ 42us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:04:58.926045 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.926117 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:58.926206 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.927297 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.879386 23569 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.004s
I20251023 02:04:58.884954 23397 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:04:58.886106 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 91106.3us
I20251023 02:04:58.922953 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 963) took 1200 ms. Trace:
I20251023 02:04:58.929921 21953 rpcz_store.cc:276] 1023 02:04:57.722633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.883023 (+1160390us) service_pool.cc:225] Handling call
1023 02:04:58.922902 (+ 39879us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.930800 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.909601 23572 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:04:58.890864 22076 leader_election.cc:336] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.508s (SENT)
I20251023 02:04:58.932425 22104 raft_consensus.cc:2410] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251023 02:04:58.933481 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:58.933907 23572 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:58.934180 22104 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:58.934783 22031 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251023 02:04:58.822937 22000 leader_election.cc:400] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:04:58.935616 22000 leader_election.cc:403] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20251023 02:04:58.936362 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 113600us
I20251023 02:04:58.885579 21957 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
W20251023 02:04:58.887140 22622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 189) took 1471 ms (client timeout 1909 ms). Trace:
I20251023 02:04:58.932056 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.937196 22000 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:58.796407 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:58.937934 21953 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
W20251023 02:04:58.937731 22622 rpcz_store.cc:269] 1023 02:04:57.416201 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:57.424784 (+ 8583us) service_pool.cc:225] Handling call
1023 02:04:58.886977 (+1462193us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b5400485154. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:58.887097 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":760260480}
I20251023 02:04:58.938236 23406 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:04:58.895519 21959 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:58.948298 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:58.949335 21959 raft_consensus.cc:2393] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:58.846546 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:58.898954 22033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 967) took 2220 ms (client timeout 2818 ms). Trace:
I20251023 02:04:58.836723 21926 service_pool.cc:138] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53732 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 963)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1250)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1211)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 933)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 98)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 99)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 100)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 938)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 948)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 949)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 958)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1246)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1257)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1258)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1260)
W20251023 02:04:58.845476 21956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1226) took 2505 ms (client timeout 3076 ms). Trace:
I20251023 02:04:58.901710 22034 raft_consensus.cc:1275] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:04:58.887300 23585 raft_consensus.cc:740] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:58.904101 22037 rpcz_store.cc:276] 1023 02:04:57.064338 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.901447 (+1837109us) service_pool.cc:225] Handling call
1023 02:04:58.903145 (+ 1698us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.903341 (+ 196us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:04:58.903365 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:58.903407 (+ 42us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:04:58.903434 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.903482 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:58.903579 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.903671 22106 raft_consensus.cc:2393] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:58.894594 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:58.910233 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:58.909122 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 103) took 2389 ms. Trace:
I20251023 02:04:58.965745 22108 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:58.910727 22036 raft_consensus.cc:1275] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:58.894330 23558 raft_consensus.cc:3055] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:04:58.900058 23637 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:58.908383 23005 consensus_queue.cc:260] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:58.894336 23404 raft_consensus.cc:697] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:58.922727 21962 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:58.813500 21961 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:58.919800 23473 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:58.937743 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 959) took 1216 ms. Trace:
I20251023 02:04:58.891705 21928 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:58.939977 21955 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:58.893744 21960 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:58.941471 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:58.939178 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:58.944761 22076 leader_election.cc:336] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.572s (SENT)
I20251023 02:04:58.946236 22000 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:58.741302 23643 consensus_queue.cc:260] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.906888 23493 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:58.947791 21925 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53734 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 977)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 972)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 967)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 973)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 974)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 964)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 965)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1250)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1207)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1209)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1211)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 933)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 98)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 99)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 100)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 938)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 948)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 949)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 958)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1246)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1257)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1258)
I20251023 02:04:58.954035 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:58.954664 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.107628 21959 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
W20251023 02:04:58.955335 22033 rpcz_store.cc:269] 1023 02:04:56.679042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.652722 (+1973680us) service_pool.cc:225] Handling call
1023 02:04:58.898909 (+246187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.965823 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1195) took 2446 ms. Trace:
I20251023 02:04:58.966645 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1207) took 2041 ms. Trace:
I20251023 02:04:58.912606 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1204) took 2298 ms. Trace:
I20251023 02:04:59.109349 22030 rpcz_store.cc:276] 1023 02:04:56.614287 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.827750 (+2213463us) service_pool.cc:225] Handling call
1023 02:04:58.872179 (+ 44429us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.912292 (+ 40113us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.912536 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:58.969404 22107 raft_consensus.cc:2410] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251023 02:04:59.111884 22111 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:59.108671 22034 rpcz_store.cc:276] 1023 02:04:56.519441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.818680 (+2299239us) service_pool.cc:225] Handling call
1023 02:04:58.844768 (+ 26088us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b58002d5a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:04:58.901304 (+ 56536us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.962190 (+ 60886us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.965650 (+ 3460us) spinlock_profiling.cc:243] Waited 56.4 ms on lock 0x7b54002ea250. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:58.965773 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":131936256}
I20251023 02:04:59.109169 22036 rpcz_store.cc:276] 1023 02:04:56.925553 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.899927 (+1974374us) service_pool.cc:225] Handling call
1023 02:04:58.910442 (+ 10515us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.966430 (+ 55988us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.966598 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.092113 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.099975 23005 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:59.100782 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.118535 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.118955 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.119583 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.102352 22076 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:59.122692 22108 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
W20251023 02:04:59.100798 21960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1230) took 2700 ms (client timeout 2672 ms). Trace:
I20251023 02:04:58.972962 23593 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:59.104228 22102 raft_consensus.cc:1275] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:59.106177 23572 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:58.970320 23558 raft_consensus.cc:740] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
W20251023 02:04:58.962131 21956 rpcz_store.cc:269] 1023 02:04:56.338073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.731663 (+2393590us) service_pool.cc:225] Handling call
1023 02:04:58.843859 (+112196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.093869 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.143520 21962 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:58.971837 21957 rpcz_store.cc:276] 1023 02:04:57.721359 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.805487 (+1084128us) service_pool.cc:225] Handling call
1023 02:04:58.937689 (+132202us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:59.123363 21960 rpcz_store.cc:269] 1023 02:04:56.400684 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.724823 (+2324139us) service_pool.cc:225] Handling call
1023 02:04:59.100701 (+375878us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:59.123123 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 178416us
I20251023 02:04:59.119511 21961 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:59.146152 22033 raft_consensus.cc:1275] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:04:59.127878 22034 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:04:59.120301 21959 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:59.148167 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:59.151768 22076 leader_election.cc:336] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.577s (SENT)
I20251023 02:04:59.100411 22103 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:59.136416 22030 raft_consensus.cc:1275] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:04:59.154228 22002 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:59.100756 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:59.146072 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.155382 21961 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:59.155457 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.156263 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.156905 21961 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:04:59.104789 23297 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251023 02:04:59.152804 22076 leader_election.cc:336] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.584s (SENT)
I20251023 02:04:59.162250 22076 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:04:59.102917 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 211266us
I20251023 02:04:59.114733 21953 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:59.164674 21928 leader_election.cc:400] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5.
W20251023 02:04:59.165226 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 221560us
I20251023 02:04:59.165567 22111 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:04:59.166890 22000 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:59.157964 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.168167 21961 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:04:59.168619 22000 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:59.169184 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:59.155453 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:04:59.170003 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:59.137199 23637 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:59.172082 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.173274 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.170861 22000 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:59.170980 22075 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:59.175122 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.177748 21960 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:04:59.141176 23643 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:59.176808 21957 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:59.142736 23572 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:59.181885 22002 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:59.189970 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:58.971850 22032 rpcz_store.cc:276] 1023 02:04:56.516511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.805157 (+2288646us) service_pool.cc:225] Handling call
1023 02:04:58.843497 (+ 38340us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.872402 (+ 28905us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:58.875496 (+ 3094us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:58.878627 (+ 3131us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:04:58.892073 (+ 13446us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:58.900826 (+ 8753us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:58.906177 (+ 5351us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.202450 21961 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:59.193503 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.179458 22111 raft_consensus.cc:1275] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:59.156354 23473 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:59.176124 23521 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:59.203651 21957 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:59.135564 23569 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.046s
I20251023 02:04:59.194097 23502 consensus_queue.cc:1059] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
W20251023 02:04:59.206512 21908 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53732 dropped due to backpressure: service queue is full with 50 items
W20251023 02:04:59.206921 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53732 dropped due to backpressure: service queue is full with 50 items
I20251023 02:04:59.207082 23502 raft_consensus.cc:3055] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:04:59.207414 23502 raft_consensus.cc:740] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:59.207511 23493 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:59.208199 23502 consensus_queue.cc:260] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:59.210052 23485 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:04:59.210412 23485 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:59.210711 23485 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:04:59.215795 21955 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:04:59.204949 22000 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:59.217422 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.218063 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.218750 21960 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
W20251023 02:04:59.221652 22002 consensus_peers.cc:597] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 -> Peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Couldn't send request to peer 02c53ba06cc3445a8534a84aa76b1c14. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53734 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
I20251023 02:04:59.222715 22002 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:59.228140 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.113955 23585 consensus_queue.cc:260] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.176041 23532 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:59.175365 22031 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251023 02:04:59.232060 23005 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:59.174806 23467 raft_consensus.cc:2749] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:04:59.194445 21954 raft_consensus.cc:2410] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20251023 02:04:59.243556 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.244470 21953 raft_consensus.cc:2410] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:04:59.244625 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:59.245857 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.181170 23630 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:59.247361 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.248409 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:59.248898 21962 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:59.230430 23520 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:59.250366 23520 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:59.250741 23520 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:59.205680 23572 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:59.217252 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:59.256107 21956 raft_consensus.cc:2368] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4.
W20251023 02:04:59.256304 23653 log.cc:927] Time spent T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20251023 02:04:59.258370 23406 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:04:59.259284 23406 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:59.259697 23406 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:59.260102 23406 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:59.263410 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.233261 22000 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:59.234500 23005 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:04:59.233227 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.257174 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.209455 23502 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:04:59.265551 23532 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.219962 23473 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.230676 23558 consensus_queue.cc:260] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:59.231554 23404 consensus_queue.cc:237] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.262795 23593 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:59.281950 23005 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:59.298883 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:59.310039 21957 raft_consensus.cc:2376] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5. Re-sending same reply.
I20251023 02:04:59.311182 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.311970 21957 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:59.313052 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.313796 21957 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
W20251023 02:04:59.301056 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 67920.7us
I20251023 02:04:59.216701 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 961) took 1494 ms. Trace:
I20251023 02:04:59.203558 23521 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:59.294912 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.245298 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.319550 21961 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:59.319793 22000 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:59.322427 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.323958 21961 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:59.325067 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.322530 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.326503 21957 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
W20251023 02:04:59.327740 22076 leader_election.cc:336] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.815s (SENT)
I20251023 02:04:59.328261 22076 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:59.308593 23558 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:59.334580 23473 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:59.314670 23593 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:04:59.335530 23655 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.173s user 0.002s sys 0.000s
I20251023 02:04:59.314241 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 964) took 2661 ms. Trace:
I20251023 02:04:59.339416 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.340076 21961 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:59.341167 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.342509 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.322156 23521 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:59.309603 21954 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:59.337246 23637 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:59.319118 21955 rpcz_store.cc:276] 1023 02:04:57.722048 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.796163 (+1074115us) service_pool.cc:225] Handling call
1023 02:04:59.216638 (+420475us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.304220 23543 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:59.345515 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:59.342952 23630 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:59.332535 21953 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:04:59.343495 21961 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:59.366904 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.367679 21953 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:59.368773 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.370563 21953 raft_consensus.cc:1275] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:59.375098 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.383131 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.363040 23297 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:04:59.385795 22110 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:04:59.386927 22109 raft_consensus.cc:1240] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:59.383877 21957 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
W20251023 02:04:59.388509 22076 leader_election.cc:336] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.971s (SENT)
I20251023 02:04:59.388967 22076 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:59.390610 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.394289 21957 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:59.392411 21962 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:59.366091 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:59.401538 22076 leader_election.cc:336] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.982s (SENT)
I20251023 02:04:59.383049 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.407881 22076 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:59.408541 21961 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 1.
I20251023 02:04:59.411224 23005 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:04:59.388027 23297 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:59.422073 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1211) took 4747 ms. Trace:
I20251023 02:04:59.422670 21962 rpcz_store.cc:276] 1023 02:04:54.674833 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.405752 (+4730919us) service_pool.cc:225] Handling call
1023 02:04:59.409870 (+ 4118us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.410251 (+ 381us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.410290 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.410350 (+ 60us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:59.410394 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.410467 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.421976 (+ 11509us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16640}
I20251023 02:04:59.374239 23630 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:59.459685 23005 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:59.384327 23485 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:04:59.460227 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 345391us
I20251023 02:04:59.447341 23532 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:59.469199 23005 raft_consensus.cc:2749] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:04:59.343516 22028 rpcz_store.cc:276] 1023 02:04:56.647714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.873185 (+2225471us) service_pool.cc:225] Handling call
1023 02:04:58.874368 (+ 1183us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:58.901929 (+ 27561us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:58.903391 (+ 1462us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:58.920071 (+ 16680us) log.cc:844] Serialized 26 byte log entry
1023 02:04:59.136106 (+216035us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:59.145879 (+ 9773us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.157935 (+ 12056us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:59.255731 (+ 97796us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:04:59.263056 (+ 7325us) raft_consensus.cc:1685] finished
1023 02:04:59.273044 (+ 9988us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.309087 (+ 36043us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27460992,"thread_start_us":197226,"threads_started":1,"wal-append.queue_time_us":242687}
I20251023 02:04:59.460199 23520 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.463008 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:59.466581 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:59.407430 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1207) took 4739 ms. Trace:
I20251023 02:04:59.482264 22109 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20251023 02:04:59.461613 22104 consensus_meta.cc:287] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.524s user 0.005s sys 0.006s
I20251023 02:04:59.458726 23493 consensus_queue.cc:1059] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20251023 02:04:59.485204 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:04:59.487327 23547 consensus_meta.cc:287] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.563s user 0.008s sys 0.001s
I20251023 02:04:59.405686 22587 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:59.415759 23473 consensus_queue.cc:1048] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20251023 02:04:59.458539 23637 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:59.447961 23437 consensus_queue.cc:1048] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
W20251023 02:04:59.465473 23491 consensus_meta.cc:287] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.578s user 0.000s sys 0.010s
W20251023 02:04:59.492343 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:59.524564 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1260) took 1103 ms. Trace:
I20251023 02:04:59.492281 22107 raft_consensus.cc:1275] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:04:59.494586 23005 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:59.466157 23585 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:59.478668 23655 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20251023 02:04:59.415740 23554 consensus_queue.cc:1048] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20251023 02:04:59.488049 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1259) took 1066 ms. Trace:
I20251023 02:04:59.496248 22104 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:59.526659 23005 raft_consensus.cc:697] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:59.536875 23005 consensus_queue.cc:237] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.540227 23520 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:59.526326 23532 raft_consensus.cc:3055] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:04:59.540828 23407 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251023 02:04:59.541673 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:59.457774 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.529906 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1212) took 4535 ms. Trace:
I20251023 02:04:59.525234 22106 rpcz_store.cc:276] 1023 02:04:58.421594 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.972719 (+551125us) service_pool.cc:225] Handling call
1023 02:04:59.110835 (+138116us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.111215 (+ 380us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:04:59.323686 (+212471us) log.cc:844] Serialized 14 byte log entry
1023 02:04:59.524177 (+200491us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.524232 (+ 55us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:59.524270 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.524344 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.524495 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":138640,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":2508,"thread_start_us":373069,"threads_started":2}
I20251023 02:04:59.494927 21953 rpcz_store.cc:276] 1023 02:04:54.668223 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.369937 (+4701714us) service_pool.cc:225] Handling call
1023 02:04:59.370249 (+ 312us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.383942 (+ 13693us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.407352 (+ 23410us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.548175 21956 rpcz_store.cc:276] 1023 02:04:54.991556 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.480968 (+4489412us) service_pool.cc:225] Handling call
1023 02:04:59.489292 (+ 8324us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.489477 (+ 185us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.489509 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.489561 (+ 52us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:59.495610 (+ 6049us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.498188 (+ 2578us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.527272 (+ 29084us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.547821 23657 raft_consensus.cc:2955] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:04:59.549182 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.549790 21953 raft_consensus.cc:1275] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:04:59.550415 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.551256 22106 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 1.
I20251023 02:04:59.501210 23547 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.533190 23642 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:04:59.545166 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 225492us
I20251023 02:04:59.553967 21962 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:04:59.550808 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1216) took 4546 ms. Trace:
I20251023 02:04:59.541690 23464 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:04:59.538923 23543 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:59.528615 23491 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:59.586880 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.588124 22106 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:59.588868 22000 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:04:59.589417 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:59.596932 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.598187 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.598893 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1263) took 1176 ms. Trace:
I20251023 02:04:59.599148 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.599313 22108 rpcz_store.cc:276] 1023 02:04:58.422491 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.123629 (+701138us) service_pool.cc:225] Handling call
1023 02:04:59.123951 (+ 322us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:59.125800 (+ 1849us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.125826 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:59.138603 (+ 12777us) log.cc:844] Serialized 26 byte log entry
1023 02:04:59.388549 (+249946us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:59.388584 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.388644 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:59.598679 (+210035us) raft_consensus.cc:1685] finished
1023 02:04:59.598700 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.598836 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13056,"thread_start_us":224127,"threads_started":1,"wal-append.queue_time_us":387022}
I20251023 02:04:59.601121 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.601323 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 98) took 4569 ms. Trace:
I20251023 02:04:59.601725 21962 rpcz_store.cc:276] 1023 02:04:55.031698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.556911 (+4525213us) service_pool.cc:225] Handling call
1023 02:04:59.600859 (+ 43948us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.601034 (+ 175us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.601062 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.601108 (+ 46us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:04:59.601136 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.601186 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.601273 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.602212 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.603096 21962 raft_consensus.cc:1275] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:59.603267 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.600379 22106 raft_consensus.cc:2410] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251023 02:04:59.603636 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1217) took 4526 ms. Trace:
I20251023 02:04:59.604786 21962 rpcz_store.cc:276] 1023 02:04:55.077408 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.602553 (+4525145us) service_pool.cc:225] Handling call
1023 02:04:59.602819 (+ 266us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.603431 (+ 612us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.603589 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.605021 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.605729 22108 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:04:59.606724 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.607452 21959 raft_consensus.cc:2393] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:04:59.607549 22108 raft_consensus.cc:2410] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20251023 02:04:59.550449 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1213) took 4556 ms. Trace:
I20251023 02:04:59.608004 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:04:59.608122 21953 rpcz_store.cc:276] 1023 02:04:54.993943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.549066 (+4555123us) service_pool.cc:225] Handling call
1023 02:04:59.549383 (+ 317us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.550160 (+ 777us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.550385 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.609640 21954 raft_consensus.cc:1275] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:59.610061 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1219) took 4295 ms. Trace:
I20251023 02:04:59.610309 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1208) took 4941 ms. Trace:
I20251023 02:04:59.610718 21954 rpcz_store.cc:276] 1023 02:04:54.668530 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.376006 (+4707476us) service_pool.cc:225] Handling call
1023 02:04:59.557439 (+181433us) spinlock_profiling.cc:243] Waited 9.29 ms on lock 0x7b5800002a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9ae20e 000000000035a3c9 00007f9c649cd030 00007f9c64a95864 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:04:59.606688 (+ 49249us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.610072 (+ 3384us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.610256 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21357952}
I20251023 02:04:59.611904 21955 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:04:59.610512 21953 rpcz_store.cc:276] 1023 02:04:55.314660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.609162 (+4294502us) service_pool.cc:225] Handling call
1023 02:04:59.609469 (+ 307us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.609663 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.609723 (+ 60us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.609764 (+ 41us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:04:59.609807 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.609883 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.610007 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.559383 21956 rpcz_store.cc:276] 1023 02:04:55.004094 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.549235 (+4545141us) service_pool.cc:225] Handling call
1023 02:04:59.549503 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.550446 (+ 943us) raft_consensus.cc:1534] Early marking committed up to index 1
1023 02:04:59.550492 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.550537 (+ 45us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:59.550572 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.550643 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.550751 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":906,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":3883}
I20251023 02:04:59.614163 21956 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:04:59.614406 21962 raft_consensus.cc:1240] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: [3.3-3.4]
I20251023 02:04:59.614645 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 99) took 3407 ms. Trace:
I20251023 02:04:59.615182 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:59.553445 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1279) took 1084 ms. Trace:
I20251023 02:04:59.614682 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 100) took 3406 ms. Trace:
I20251023 02:04:59.616011 22106 raft_consensus.cc:2368] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5.
I20251023 02:04:59.616204 21956 rpcz_store.cc:276] 1023 02:04:56.207838 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.613568 (+3405730us) service_pool.cc:225] Handling call
1023 02:04:59.613884 (+ 316us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.614471 (+ 587us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.614634 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.616892 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.617771 22106 raft_consensus.cc:2393] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:59.615965 22102 rpcz_store.cc:276] 1023 02:04:58.469432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.259333 (+789901us) service_pool.cc:225] Handling call
1023 02:04:59.259626 (+ 293us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.259826 (+ 200us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:04:59.260121 (+ 295us) log.cc:844] Serialized 14 byte log entry
1023 02:04:59.532878 (+272757us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.536194 (+ 3316us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:04:59.538643 (+ 2449us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.543099 (+ 4456us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.551707 (+ 8608us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":548,"prepare.queue_time_us":12709,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":6112,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":2689,"thread_start_us":248951,"threads_started":1,"wal-append.queue_time_us":304885}
I20251023 02:04:59.618076 23485 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:59.618461 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.544291 23532 raft_consensus.cc:740] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:04:59.619549 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 938) took 3411 ms. Trace:
I20251023 02:04:59.620452 22000 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:59.621523 21962 rpcz_store.cc:276] 1023 02:04:56.208154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.613592 (+3405438us) service_pool.cc:225] Handling call
1023 02:04:59.614068 (+ 476us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:04:59.614728 (+ 660us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.619503 (+ 4775us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.615053 21953 rpcz_store.cc:276] 1023 02:04:56.207477 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.613881 (+3406404us) service_pool.cc:225] Handling call
1023 02:04:59.614165 (+ 284us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.614344 (+ 179us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.614371 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.614415 (+ 44us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:04:59.614446 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.614500 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.614596 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.607095 23491 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:59.607930 23547 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:59.624150 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.630501 22035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 106) took 3113 ms. Trace:
I20251023 02:04:59.618882 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:59.640997 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:59.618063 21956 raft_consensus.cc:1275] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251023 02:04:59.631335 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 229852us
W20251023 02:04:59.664498 21925 connection.cc:232] RPC call timeout handler was delayed by -2.31425494899986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26880000000000004s for a response
W20251023 02:04:59.665740 21925 connection.cc:232] RPC call timeout handler was delayed by -2.0476783309999291s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
W20251023 02:04:59.666219 22076 leader_election.cc:336] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.532s (SENT)
W20251023 02:04:59.667011 22076 leader_election.cc:336] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.570s (SENT)
I20251023 02:04:59.667230 22076 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:59.659376 21962 raft_consensus.cc:1275] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:59.648825 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1223) took 3322 ms. Trace:
I20251023 02:04:59.686132 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.687146 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1222) took 3361 ms. Trace:
I20251023 02:04:59.687587 21962 rpcz_store.cc:276] 1023 02:04:56.325902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.632254 (+3306352us) service_pool.cc:225] Handling call
1023 02:04:59.659079 (+ 26825us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.669167 (+ 10088us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.687080 (+ 17913us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:59.651921 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 429297us
I20251023 02:04:59.652808 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:59.653534 23520 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:59.654057 23485 consensus_queue.cc:1048] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:59.654757 23541 consensus_queue.cc:1048] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:04:59.644440 22035 rpcz_store.cc:276] 1023 02:04:56.517392 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.818374 (+2300982us) service_pool.cc:225] Handling call
1023 02:04:58.840288 (+ 21914us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.840512 (+ 224us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:04:58.958118 (+117606us) log.cc:844] Serialized 14 byte log entry
1023 02:04:59.334967 (+376849us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:04:59.390688 (+ 55721us) log.cc:844] Serialized 14 byte log entry
1023 02:04:59.630076 (+239388us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.630142 (+ 66us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:59.630198 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.630284 (+ 86us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.630434 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67025,"prepare.queue_time_us":309500,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":68,"raft.queue_time_us":220392,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":22741,"spinlock_wait_cycles":29068032,"thread_start_us":583336,"threads_started":3,"wal-append.queue_time_us":203971}
I20251023 02:04:59.620448 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.660341 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1225) took 3329 ms. Trace:
I20251023 02:04:59.646327 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 128) took 1081 ms. Trace:
I20251023 02:04:59.660008 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1265) took 1237 ms. Trace:
I20251023 02:04:59.662917 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:59.677641 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 258991us
I20251023 02:04:59.679654 23464 raft_consensus.cc:697] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:04:59.680414 22076 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:59.609417 23406 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.690903 22002 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:59.692636 22108 raft_consensus.cc:2393] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:59.694125 21925 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:04:59.696074 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:59.720511 22107 rpcz_store.cc:276] 1023 02:04:58.423021 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.134967 (+711946us) service_pool.cc:225] Handling call
1023 02:04:59.141452 (+ 6485us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.525794 (+384342us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.659798 (+134004us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b54000118d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:59.659953 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":806170368}
I20251023 02:04:59.721252 21959 rpcz_store.cc:276] 1023 02:04:56.331064 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.659436 (+3328372us) service_pool.cc:225] Handling call
1023 02:04:59.659763 (+ 327us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.659957 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.659994 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.660048 (+ 54us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:59.660095 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.660154 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.660273 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.721917 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.719084 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.722121 22109 rpcz_store.cc:276] 1023 02:04:58.564733 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.460073 (+895340us) service_pool.cc:225] Handling call
1023 02:04:59.478812 (+ 18739us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.495152 (+ 16340us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.646062 (+150910us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b54000fda54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:04:59.646259 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7044992}
I20251023 02:04:59.723870 21957 raft_consensus.cc:1240] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:59.658540 23554 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:59.806645 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1209) took 5137 ms. Trace:
W20251023 02:04:59.806952 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 116147us
I20251023 02:04:59.755421 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.807243 21957 rpcz_store.cc:276] 1023 02:04:54.668841 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.405643 (+4736802us) service_pool.cc:225] Handling call
1023 02:04:59.455978 (+ 50335us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b5800002a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9ae20e 000000000035a3c9 00007f9c649cd030 00007f9c64a95864 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:04:59.456162 (+ 184us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.723655 (+267493us) spinlock_profiling.cc:243] Waited 69.3 ms on lock 0x7b54003866a0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa10d10 00007f9c5fa9ad50 00007f9c5fa9bdc1 00007f9c5fa9892f 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c
1023 02:04:59.806398 (+ 82743us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.806593 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":165904512}
I20251023 02:04:59.809024 22002 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:59.810277 22002 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:59.690061 23491 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:04:59.806985 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.684288 23385 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:04:59.759349 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:59.812441 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 160629us
I20251023 02:04:59.818565 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.821080 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:04:59.813381 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 133047us
I20251023 02:04:59.821985 21962 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:59.812059 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.822654 22076 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:59.812342 23005 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:59.823599 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.823909 23005 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:59.824260 21958 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:04:59.824599 23005 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:04:59.821707 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 127734us
I20251023 02:04:59.812261 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:04:59.835971 21957 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:59.812783 23404 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:04:59.837260 21957 raft_consensus.cc:1275] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251023 02:04:59.827116 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 129) took 1247 ms. Trace:
I20251023 02:04:59.744099 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 948) took 3403 ms. Trace:
I20251023 02:04:59.839355 22103 rpcz_store.cc:276] 1023 02:04:58.579547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.460446 (+880899us) service_pool.cc:225] Handling call
1023 02:04:59.462693 (+ 2247us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:59.463057 (+ 364us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.463092 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:59.463566 (+ 474us) log.cc:844] Serialized 26 byte log entry
1023 02:04:59.679794 (+216228us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:04:59.679841 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.679926 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:59.826880 (+146954us) raft_consensus.cc:1685] finished
1023 02:04:59.826909 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.827048 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1383,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":14280,"thread_start_us":215812,"threads_started":1,"wal-append.queue_time_us":356705}
I20251023 02:04:59.839640 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.828761 23464 consensus_queue.cc:237] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:59.829470 23669 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:59.830801 23406 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:59.823915 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:59.831285 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.842751 23669 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:59.831627 23637 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:59.843113 21960 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:04:59.843223 23669 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:59.843410 23637 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:59.843976 23637 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:04:59.844720 23558 consensus_meta.cc:287] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.505s user 0.001s sys 0.006s
I20251023 02:04:59.831892 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:04:59.845515 21962 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:59.843255 22032 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:59.743635 23437 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:59.846743 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:59.846560 23437 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:59.847855 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.820935 23674 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:59.849175 23674 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:04:59.849548 23674 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:59.850306 22032 raft_consensus.cc:1240] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20251023 02:04:59.834064 23670 log.cc:927] Time spent T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20251023 02:04:59.813169 23593 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.854735 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1278) took 1385 ms. Trace:
I20251023 02:04:59.855907 23593 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:59.856312 21928 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:59.855113 22111 rpcz_store.cc:276] 1023 02:04:58.469148 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.222310 (+753162us) service_pool.cc:225] Handling call
1023 02:04:59.243996 (+ 21686us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:59.244717 (+ 721us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.244769 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:59.334140 (+ 89371us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b2c0005f718. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5e3949b1 00007f9c5e3a3fe2 00007f9c5fad4701 00007f9c5fa98fda 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22
1023 02:04:59.334521 (+ 381us) log.cc:844] Serialized 26 byte log entry
1023 02:04:59.531918 (+197397us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:59.531973 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.532071 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:59.854415 (+322344us) raft_consensus.cc:1685] finished
1023 02:04:59.854438 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.854679 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"spinlock_wait_cycles":23044864,"thread_start_us":196934,"threads_started":1,"wal-append.queue_time_us":285712}
W20251023 02:04:59.856887 23659 log.cc:927] Time spent T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.213s user 0.000s sys 0.000s
I20251023 02:04:59.812841 23554 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.858693 22032 raft_consensus.cc:1240] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251023 02:04:59.859380 23554 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:59.691162 23543 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:04:59.532914 22105 rpcz_store.cc:276] 1023 02:04:58.421286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.937611 (+516325us) service_pool.cc:225] Handling call
1023 02:04:58.937925 (+ 314us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:58.938145 (+ 220us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:04:59.031798 (+ 93653us) log.cc:844] Serialized 14 byte log entry
1023 02:04:59.416583 (+384785us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.454355 (+ 37772us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:59.458607 (+ 4252us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.464446 (+ 5839us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.487980 (+ 23534us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":311483,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":9438,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":11633,"thread_start_us":321336,"threads_started":2,"wal-append.queue_time_us":435955}
I20251023 02:04:59.876789 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1289) took 1220 ms. Trace:
I20251023 02:04:59.823396 23491 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.879441 23491 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:04:59.834619 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.881318 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.882910 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.835232 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.658735 23661 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:04:59.884820 23661 raft_consensus.cc:697] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:04:59.885764 23661 consensus_queue.cc:237] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.838697 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1221) took 4395 ms. Trace:
I20251023 02:04:59.877275 22105 rpcz_store.cc:276] 1023 02:04:58.656177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.875750 (+1219573us) service_pool.cc:225] Handling call
1023 02:04:59.876142 (+ 392us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.876362 (+ 220us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.876402 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.876458 (+ 56us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:59.876512 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.876591 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.876723 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.839891 21956 rpcz_store.cc:276] 1023 02:04:56.322825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.617423 (+3294598us) service_pool.cc:225] Handling call
1023 02:04:59.617766 (+ 343us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.684404 (+ 66638us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.726474 (+ 42070us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.841913 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.810137 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.889425 22036 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:04:59.845041 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1233) took 3426 ms. Trace:
I20251023 02:04:59.890895 21960 rpcz_store.cc:276] 1023 02:04:56.419079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.844292 (+3425213us) service_pool.cc:225] Handling call
1023 02:04:59.844561 (+ 269us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.844744 (+ 183us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.844768 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.844809 (+ 41us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:59.844841 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.844896 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.844991 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.887902 21954 rpcz_store.cc:276] 1023 02:04:55.443609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.611620 (+4168011us) service_pool.cc:225] Handling call
1023 02:04:59.611960 (+ 340us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:59.612331 (+ 371us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.612360 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:59.612856 (+ 496us) log.cc:844] Serialized 26 byte log entry
1023 02:04:59.681769 (+ 68913us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:59.681831 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.681922 (+ 91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:59.838503 (+156581us) raft_consensus.cc:1685] finished
1023 02:04:59.838530 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.838645 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4094,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":68349,"threads_started":1,"wal-append.queue_time_us":221878}
I20251023 02:04:59.892643 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.893064 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1234) took 3428 ms. Trace:
I20251023 02:04:59.893551 22107 raft_consensus.cc:2393] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:59.893522 21960 rpcz_store.cc:276] 1023 02:04:56.464548 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.892187 (+3427639us) service_pool.cc:225] Handling call
1023 02:04:59.892484 (+ 297us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.892689 (+ 205us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.892725 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.892779 (+ 54us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:59.892822 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.892899 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.893004 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:04:59.894959 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:04:59.847961 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1210) took 1238 ms. Trace:
I20251023 02:04:59.850546 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:04:59.698256 23532 consensus_queue.cc:260] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.861546 23669 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:59.743955 23630 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:59.862716 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1228) took 3467 ms. Trace:
I20251023 02:04:59.786732 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:04:59.864530 23585 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:04:59.865095 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.914080 22034 raft_consensus.cc:2393] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:59.818101 21953 raft_consensus.cc:1275] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251023 02:04:59.826414 23672 log.cc:927] Time spent T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20251023 02:04:59.833118 21925 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:59.920141 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 87101.3us
I20251023 02:04:59.921137 21925 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:04:59.832597 22076 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:04:59.922477 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 89957.4us
I20251023 02:04:59.923450 22076 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:04:59.924827 22076 leader_election.cc:400] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4.
I20251023 02:04:59.925136 22076 leader_election.cc:403] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20251023 02:04:59.866096 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:04:59.867605 23682 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20251023 02:04:59.867707 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.931222 22106 raft_consensus.cc:2410] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251023 02:04:59.862322 23637 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.868249 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 131) took 1287 ms. Trace:
I20251023 02:04:59.936952 23585 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:04:59.937278 23467 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4.
I20251023 02:04:59.917126 23593 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20251023 02:04:59.868691 23407 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:59.868270 23547 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:04:59.870146 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1287) took 1214 ms. Trace:
I20251023 02:04:59.870214 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 972) took 1332 ms. Trace:
I20251023 02:04:59.871280 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 130) took 1291 ms. Trace:
I20251023 02:04:59.937551 23585 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:04:59.866191 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 475807us
I20251023 02:04:59.940966 23585 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:04:59.938269 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 949) took 3601 ms. Trace:
I20251023 02:04:59.942128 21953 rpcz_store.cc:276] 1023 02:04:56.337412 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.664363 (+3326951us) service_pool.cc:225] Handling call
1023 02:04:59.758600 (+ 94237us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.920135 (+161535us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.938194 (+ 18059us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.943552 22587 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:59.940095 23547 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:04:59.944375 23547 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:04:59.845839 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.945497 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:59.946080 23637 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:04:59.946575 23642 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:04:59.872187 23437 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:04:59.872395 21959 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:04:59.940339 22037 rpcz_store.cc:276] 1023 02:04:58.537844 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.971056 (+433212us) service_pool.cc:225] Handling call
1023 02:04:58.971247 (+ 191us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:59.106654 (+135407us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.106688 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:59.107123 (+ 435us) log.cc:844] Serialized 26 byte log entry
1023 02:04:59.402274 (+295151us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:59.402320 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.402383 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:04:59.869960 (+467577us) raft_consensus.cc:1685] finished
1023 02:04:59.869989 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.870145 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":164643,"raft.queue_time_us":136894,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":20449,"thread_start_us":244243,"threads_started":1,"wal-append.queue_time_us":432865}
I20251023 02:04:59.957808 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:04:59.872999 21928 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:04:59.887573 22033 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251023 02:04:59.690318 23297 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.888785 23005 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:04:59.685083 21955 rpcz_store.cc:276] 1023 02:04:56.326145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.647669 (+3321524us) service_pool.cc:225] Handling call
1023 02:04:59.648081 (+ 412us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.648314 (+ 233us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.648355 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.648445 (+ 90us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:59.648495 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.648578 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.648755 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.758127 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.697386 23520 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:04:59.910601 21958 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33"
dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:40860
I20251023 02:04:59.910730 21957 rpcz_store.cc:276] 1023 02:04:56.395201 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.836672 (+3441471us) service_pool.cc:225] Handling call
1023 02:04:59.836949 (+ 277us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.862440 (+ 25491us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.862639 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.981380 21958 raft_consensus.cc:606] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:59.981900 21958 raft_consensus.cc:614] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:04:59.982309 21957 raft_consensus.cc:1275] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:04:59.983067 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1258) took 1225 ms. Trace:
I20251023 02:04:59.983448 21957 rpcz_store.cc:276] 1023 02:04:58.757184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.981513 (+1224329us) service_pool.cc:225] Handling call
1023 02:04:59.981911 (+ 398us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.982787 (+ 876us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.983003 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.985050 21957 raft_consensus.cc:1275] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:59.864045 21962 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:04:59.986555 21962 raft_consensus.cc:1240] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:04:59.914278 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1235) took 3418 ms. Trace:
I20251023 02:04:59.988267 21954 rpcz_store.cc:276] 1023 02:04:56.496179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.892494 (+3396315us) service_pool.cc:225] Handling call
1023 02:04:59.913851 (+ 21357us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.913998 (+ 147us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.914022 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.914060 (+ 38us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:59.914086 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.914135 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.914229 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.723047 23541 consensus_queue.cc:1243] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer ca1362b9d9b34073bf35b6f6c9d1159a log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251023 02:04:59.916281 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 958) took 3213 ms. Trace:
I20251023 02:04:59.996551 21960 rpcz_store.cc:276] 1023 02:04:56.703088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.894699 (+3191611us) service_pool.cc:225] Handling call
1023 02:04:59.895017 (+ 318us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.909928 (+ 14911us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.909973 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.912056 (+ 2083us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:04:59.916024 (+ 3968us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.916098 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.916232 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":14671,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":7499}
I20251023 02:04:59.753258 23521 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:04:59.999820 23005 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:04:59.911998 23572 consensus_meta.cc:287] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.657s user 0.013s sys 0.000s
I20251023 02:05:00.005946 21956 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:04:59.926699 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:04:59.909476 23532 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:59.927552 22030 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:04:59.931871 21925 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:00.015554 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 85117us
I20251023 02:04:59.979930 22102 raft_consensus.cc:2393] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:04:59.907219 22029 rpcz_store.cc:276] 1023 02:04:58.609392 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.725798 (+116406us) service_pool.cc:225] Handling call
1023 02:04:59.847758 (+1121960us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b540040c9d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:04:59.847885 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1715195264}
I20251023 02:04:59.933439 22076 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:04:59.940214 22111 rpcz_store.cc:276] 1023 02:04:58.655603 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.857075 (+1201472us) service_pool.cc:225] Handling call
1023 02:04:59.869446 (+ 12371us) spinlock_profiling.cc:243] Waited 9.11 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:04:59.869707 (+ 261us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.869866 (+ 159us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.869887 (+ 21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.869929 (+ 42us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:04:59.869963 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.870011 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.870099 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20957440}
I20251023 02:04:59.940444 22108 rpcz_store.cc:276] 1023 02:04:58.580243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.865467 (+1285224us) service_pool.cc:225] Handling call
1023 02:04:59.870831 (+ 5364us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.871004 (+ 173us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.871029 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.871071 (+ 42us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:04:59.871100 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.871150 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.871232 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1525632}
W20251023 02:05:00.019769 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 86430.9us
I20251023 02:05:00.020659 22076 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.020959 22108 raft_consensus.cc:1275] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:00.022646 22108 raft_consensus.cc:1275] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:04:59.960203 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1298) took 1120 ms. Trace:
W20251023 02:04:59.960913 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 88019.1us
W20251023 02:04:59.962427 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 273951us
I20251023 02:05:00.024713 21928 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:00.024297 22106 rpcz_store.cc:276] 1023 02:04:58.839428 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.959279 (+1119851us) service_pool.cc:225] Handling call
1023 02:04:59.959603 (+ 324us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.959807 (+ 204us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.959843 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.959897 (+ 54us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:04:59.959950 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.960022 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.960139 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:04:59.967306 22035 raft_consensus.cc:2393] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:00.026947 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.027806 22035 raft_consensus.cc:2393] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:04:59.969339 22036 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:04:59.929325 23543 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:04:59.939924 23407 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:59.976389 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.978209 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 135) took 1207 ms. Trace:
I20251023 02:05:00.030941 22105 rpcz_store.cc:276] 1023 02:04:58.771257 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.912519 (+1141262us) service_pool.cc:225] Handling call
1023 02:04:59.957438 (+ 44919us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.977767 (+ 20329us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.977811 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.977873 (+ 62us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:04:59.977917 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.978007 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.978144 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:00.031756 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:04:59.936134 22103 rpcz_store.cc:276] 1023 02:04:58.580836 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.841100 (+1260264us) service_pool.cc:225] Handling call
1023 02:04:59.867536 (+ 26436us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:04:59.867751 (+ 215us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.867925 (+ 174us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.867951 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:04:59.868002 (+ 51us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:04:59.868032 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.868103 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:04:59.868200 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57676160}
I20251023 02:05:00.016341 21925 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:04:59.934262 23473 raft_consensus.cc:2804] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:04:59.997360 23520 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:04:59.974083 23437 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:04:59.968391 23674 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.021796 22076 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.037331 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.039812 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 126) took 1476 ms. Trace:
I20251023 02:05:00.051249 22110 rpcz_store.cc:276] 1023 02:04:58.563537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.387028 (+823491us) service_pool.cc:225] Handling call
1023 02:04:59.387392 (+ 364us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:04:59.392353 (+ 4961us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:04:59.392391 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:04:59.416695 (+ 24304us) log.cc:844] Serialized 26 byte log entry
1023 02:04:59.676361 (+259666us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:04:59.676408 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:04:59.676492 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:00.039560 (+363068us) raft_consensus.cc:1685] finished
1023 02:05:00.039587 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:00.039747 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6270,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":9653,"thread_start_us":212091,"threads_started":1,"wal-append.queue_time_us":287538}
I20251023 02:05:00.040930 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.042565 21925 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.048509 22111 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251023 02:05:00.057262 22076 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.058437 22076 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:00.028724 22108 raft_consensus.cc:1240] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 1. Current term is 2. Ops: []
I20251023 02:05:00.024660 23554 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:05:00.055961 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.030375 23642 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:00.036218 23630 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:00.030175 23491 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:05:00.046069 23437 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:00.066869 23467 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:00.068197 23437 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:00.068586 23467 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:00.069141 23467 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.041705 23473 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:00.069924 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 259746us
I20251023 02:05:00.070158 23473 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:00.071105 22002 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:00.043533 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 100063us
I20251023 02:05:00.064756 22076 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.017508 23572 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.060848 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:00.035930 23404 consensus_queue.cc:1059] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20251023 02:05:00.047676 23502 consensus_queue.cc:1059] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20251023 02:05:00.066375 23630 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:00.087543 23404 raft_consensus.cc:3055] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:00.090025 21961 raft_consensus.cc:1240] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:00.091147 23572 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.094905 22109 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20251023 02:05:00.095254 23297 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:00.096416 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:00.144140 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 195654us
I20251023 02:05:00.098400 23521 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.099184 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.086361 23554 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:00.101233 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 952) took 3641 ms. Trace:
I20251023 02:05:00.159502 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 933) took 5201 ms. Trace:
I20251023 02:05:00.064441 22032 raft_consensus.cc:1240] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251023 02:05:00.064728 23397 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:05:00.063112 23637 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:00.066028 23543 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251023 02:05:00.111341 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 85377.6us
W20251023 02:05:00.118010 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 88129.9us
I20251023 02:05:00.146445 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.116914 23404 raft_consensus.cc:740] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:00.155529 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:00.160141 21961 rpcz_store.cc:276] 1023 02:04:54.957781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.410322 (+4452541us) service_pool.cc:225] Handling call
1023 02:04:59.410651 (+ 329us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:00.157705 (+747054us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:00.159166 (+ 1461us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b5400107f54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:00.159397 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1562137344}
I20251023 02:05:00.159933 21956 rpcz_store.cc:276] 1023 02:04:56.460262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.889267 (+3429005us) service_pool.cc:225] Handling call
1023 02:04:59.889545 (+ 278us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:00.006495 (+116950us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:00.101018 (+ 94523us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54002dcdd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:00.101178 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":267148032}
W20251023 02:05:00.161864 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 90826.6us
I20251023 02:05:00.185880 22002 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.079772 23491 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:00.190915 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:00.166419 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 123953us
I20251023 02:05:00.096055 23547 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.175810 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:00.197394 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1257) took 1440 ms. Trace:
I20251023 02:05:00.116629 23630 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:00.181860 21928 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:00.182396 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 117719us
I20251023 02:05:00.107168 23464 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:00.193567 21925 leader_election.cc:400] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5.
I20251023 02:05:00.202034 22076 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.202239 21928 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:00.202432 21925 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.203191 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.080780 23679 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:05:00.197957 21959 rpcz_store.cc:276] 1023 02:04:58.756871 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.956090 (+1199219us) service_pool.cc:225] Handling call
1023 02:04:59.956437 (+ 347us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:04:59.956690 (+ 253us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:05:00.043752 (+ 87062us) log.cc:844] Serialized 14 byte log entry
1023 02:05:00.196962 (+153210us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:00.197033 (+ 71us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:00.197091 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:00.197179 (+ 88us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:00.197330 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":109018,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":53,"thread_start_us":221495,"threads_started":2}
I20251023 02:05:00.207847 21925 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.174710 23637 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:00.165956 23669 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:00.180261 23541 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:00.210968 21925 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.185129 23642 raft_consensus.cc:697] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:00.181836 23543 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:00.212049 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.212464 23593 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:00.218147 23543 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:00.218327 23593 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:00.211396 22002 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.209095 21928 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:00.219753 21928 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.180900 23297 raft_consensus.cc:3055] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:00.218395 22037 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:00.186928 23684 consensus_queue.cc:1059] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20251023 02:05:00.209043 23572 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:00.223433 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.223722 22002 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:00.211148 23491 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:00.210520 23541 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:00.211897 23637 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.242794 22002 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:00.187232 23558 consensus_queue.cc:1059] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.028s
I20251023 02:05:00.250391 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.245504 22002 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.215060 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.257172 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.256265 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.180011 23688 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20251023 02:05:00.218914 23593 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.219209 23554 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:00.216738 23397 raft_consensus.cc:697] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:00.268124 22002 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.268934 23005 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:00.269286 23005 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:00.269569 23005 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.221894 23297 raft_consensus.cc:740] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:00.228581 23669 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:00.210603 23464 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:00.254112 23541 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:00.264003 23693 log.cc:927] Time spent T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20251023 02:05:00.277274 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:00.276739 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 74763.1us
I20251023 02:05:00.278738 22076 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.258402 23572 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:00.265964 23691 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:00.266443 23661 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:00.290572 23691 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:00.290683 23661 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:00.291085 23691 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:00.219599 23404 consensus_queue.cc:260] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.292152 23404 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:00.279651 23687 consensus_queue.cc:1048] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.011s
I20251023 02:05:00.276705 23674 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:00.283869 23675 consensus_queue.cc:1048] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:00.294795 23464 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.283126 23297 consensus_queue.cc:260] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.295509 23297 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:00.296006 23705 log.cc:927] Time spent T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20251023 02:05:00.296931 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:00.280982 23554 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:00.271342 22029 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:05:00.285138 23437 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.289871 23703 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:00.265697 23671 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:00.291177 23661 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.269958 23491 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:00.297523 22030 raft_consensus.cc:2376] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5. Re-sending same reply.
I20251023 02:05:00.286087 23406 consensus_queue.cc:1048] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:05:00.286969 23005 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.298890 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.299711 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:00.326439 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 106813us
I20251023 02:05:00.272819 23669 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:00.303758 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1295) took 1584 ms. Trace:
I20251023 02:05:00.304930 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:00.276083 23679 raft_consensus.cc:3055] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:00.314913 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:00.327387 21928 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:00.316010 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:00.326073 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:00.287191 23547 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.329710 22107 rpcz_store.cc:276] 1023 02:04:58.719347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.906121 (+1186774us) service_pool.cc:225] Handling call
1023 02:04:59.915006 (+ 8885us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:04:59.922346 (+ 7340us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:04:59.922393 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:04:59.953392 (+ 30999us) log.cc:844] Serialized 52 byte log entry
1023 02:05:00.163741 (+210349us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:00.205825 (+ 42084us) log.cc:844] Serialized 14 byte log entry
1023 02:05:00.277222 (+ 71397us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:00.277295 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:00.303536 (+ 26241us) raft_consensus.cc:1685] finished
1023 02:05:00.303558 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:00.303704 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":326,"raft.queue_time_us":473,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":9754,"thread_start_us":253096,"threads_started":3,"wal-append.queue_time_us":290675}
I20251023 02:05:00.331912 23005 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.336710 22105 raft_consensus.cc:2393] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
W20251023 02:05:00.337677 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 126795us
I20251023 02:05:00.333252 22106 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:00.338452 21925 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.339008 21928 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.305636 23703 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:00.335095 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 56394.7us
I20251023 02:05:00.341151 21925 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.341066 22002 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:00.342206 21928 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.380798 22076 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.326854 23674 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:00.335513 22033 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:00.385946 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 44885.6us
W20251023 02:05:00.385878 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 43773.7us
W20251023 02:05:00.386233 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 45399.8us
I20251023 02:05:00.386942 21925 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.331418 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:00.305254 23673 consensus_queue.cc:1048] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.099s
I20251023 02:05:00.389823 23532 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:00.303742 23642 consensus_queue.cc:237] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.340077 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:00.391232 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:00.411827 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:00.333047 23679 raft_consensus.cc:740] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:00.332158 23669 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:00.394203 22076 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.344686 23572 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:00.344946 23491 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:00.332103 23709 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:05:00.377694 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.335784 23554 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:00.396050 21925 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.454840 22106 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:00.460047 22106 raft_consensus.cc:1240] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:00.461314 22106 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:00.451717 23717 raft_consensus.cc:2804] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:05:00.462004 23717 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:00.462354 23717 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:00.463189 23679 consensus_queue.cc:260] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:00.464375 23679 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:00.466444 23437 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.468470 23473 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.435678 23532 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:05:00.471316 23585 consensus_meta.cc:287] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.530s user 0.009s sys 0.000s
I20251023 02:05:00.471722 23585 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.474925 23467 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.455621 23704 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:00.476848 23467 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:00.476625 23704 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:00.479171 23704 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.481180 22110 raft_consensus.cc:1240] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:00.380267 23703 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.387836 23385 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:00.384797 23674 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:00.449995 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 93681.7us
I20251023 02:05:00.469518 21953 raft_consensus.cc:1240] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
W20251023 02:05:00.531941 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 82405.8us
I20251023 02:05:00.452751 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:00.437004 23721 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:00.488019 22000 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:00.488253 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 92294.8us
I20251023 02:05:00.432053 23710 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:00.434214 23720 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:00.477451 23467 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:00.428956 23675 consensus_queue.cc:1048] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
W20251023 02:05:00.530654 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:00.483937 22033 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:00.535072 21925 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.483279 23709 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:00.532922 23674 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.547382 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:00.547616 21925 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.544133 22077 leader_election.cc:400] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4.
W20251023 02:05:00.547055 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 152950us
I20251023 02:05:00.549047 22077 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:00.549880 22076 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.540505 23710 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.390216 23397 consensus_queue.cc:237] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:00.528951 23491 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:00.534454 23585 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.550738 22076 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.540611 23467 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:00.554381 21925 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.533994 23721 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:00.550531 22077 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:00.545411 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 57456.7us
W20251023 02:05:00.550033 22589 leader_election.cc:336] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.759s (SENT)
I20251023 02:05:00.585827 22589 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:00.553690 23637 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:00.557345 23520 consensus_meta.cc:287] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.508s user 0.008s sys 0.000s
I20251023 02:05:00.531292 23407 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.546865 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:00.558158 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:00.558554 23297 raft_consensus.cc:2749] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:00.539822 23673 consensus_queue.cc:1048] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20251023 02:05:00.560964 23716 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:00.567878 23661 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.546847 23709 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:00.573274 23473 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.573390 23543 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.579280 22076 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.581650 21925 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.527804 23541 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.588919 22000 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.552843 23720 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:00.559127 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:00.598584 22102 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.598675 23725 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:05:00.620472 23716 raft_consensus.cc:2804] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:05:00.580183 23689 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:00.555030 23406 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.044s
I20251023 02:05:00.552326 23630 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.584017 23585 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:00.625224 23716 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:00.591066 23657 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:05:00.613471 22076 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:00.591943 23721 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.565363 23593 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.583287 23467 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.580123 23713 consensus_queue.cc:1048] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20251023 02:05:00.596536 23675 consensus_queue.cc:1059] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20251023 02:05:00.599368 23687 consensus_queue.cc:1059] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20251023 02:05:00.629007 22036 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:00.617954 23297 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:00.596976 23520 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:00.582513 23717 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.608086 21925 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.622720 23521 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:00.650216 22034 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:00.638897 22108 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:00.635397 23716 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:05:00.640530 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 106967us
I20251023 02:05:00.631980 23642 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:00.631865 23689 raft_consensus.cc:697] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:00.633234 23725 raft_consensus.cc:697] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:00.630656 23727 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:00.633373 23729 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:00.678148 21960 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:05:00.636351 23703 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.654647 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.655470 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.683678 21958 raft_consensus.cc:2410] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251023 02:05:00.684048 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.684619 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.684844 21959 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.657292 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.657934 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.685722 21961 raft_consensus.cc:2368] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:05:00.686703 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.661696 22589 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:00.633646 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 83218.4us
I20251023 02:05:00.674139 23709 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.680765 22076 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:00.690869 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.691051 21957 raft_consensus.cc:2410] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251023 02:05:00.691431 21954 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:00.691890 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:00.692206 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.692589 21957 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:00.680399 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.700285 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.794719 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:00.750198 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:00.785656 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.788187 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.690701 23657 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:00.651484 23297 raft_consensus.cc:2749] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:05:00.805135 21962 raft_consensus.cc:2393] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:00.792946 21956 raft_consensus.cc:2393] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:00.798908 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:00.807257 21958 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:00.808399 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:00.803907 21961 raft_consensus.cc:2393] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:00.814702 21959 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:00.785403 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.785925 23456 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:05:00.821798 21958 raft_consensus.cc:2393] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:00.829782 23732 raft_consensus.cc:2764] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251023 02:05:00.830060 23675 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:00.832450 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:00.833724 21957 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:00.839812 21954 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:00.675554 23642 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:00.846246 23728 log.cc:927] Time spent T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20251023 02:05:00.847391 23730 log.cc:927] Time spent T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.197s user 0.000s sys 0.000s
I20251023 02:05:00.847903 23385 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:00.848477 23642 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:00.785571 23464 consensus_queue.cc:1059] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20251023 02:05:00.807113 23657 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:00.849400 23689 consensus_queue.cc:237] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.851763 23657 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.841828 21955 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:00.800319 23407 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:00.818447 23691 consensus_meta.cc:287] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.527s user 0.009s sys 0.001s
I20251023 02:05:00.694192 23729 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.801357 23733 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:00.827765 23710 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:00.831939 21958 raft_consensus.cc:2955] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:00.830533 23675 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:00.830742 23732 raft_consensus.cc:2764] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251023 02:05:00.837926 23456 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:00.830590 23637 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:00.855115 21962 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "93b201674548485eaa29b78067f74a66"
dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:40860
I20251023 02:05:00.803126 23679 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20251023 02:05:00.868800 23691 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:00.861171 23733 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:00.875953 23733 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:00.876619 23691 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:00.842828 23661 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.876976 23630 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:00.878444 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:00.853886 23541 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.870167 23456 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:00.917862 21962 raft_consensus.cc:606] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:00.889868 23725 consensus_queue.cc:237] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.866657 22104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 136) took 1934 ms. Trace:
I20251023 02:05:00.879086 23297 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:00.927042 23593 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:00.878274 23543 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:00.939862 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 259228us
I20251023 02:05:00.940485 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:00.941360 21962 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:05:00.942687 22589 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
W20251023 02:05:00.947659 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 259784us
W20251023 02:05:00.954239 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:00.926420 23520 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:00.960752 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:00.960666 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:00.961922 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.962184 22030 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:00.962662 22032 raft_consensus.cc:1240] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:00.963111 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:00.970875 22033 raft_consensus.cc:2393] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:00.963587 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:00.964042 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:00.934180 22618 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:05:00.939913 23717 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:00.975896 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:00.987922 22035 raft_consensus.cc:2393] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:00.950913 23704 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:00.992661 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.980026 22076 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:00.998003 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:00.999047 22590 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:00.937602 23585 consensus_queue.cc:1059] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20251023 02:05:01.004295 22037 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:01.006786 22037 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251023 02:05:00.997190 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:00.974278 23703 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:01.011042 23541 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:01.011525 22028 raft_consensus.cc:2393] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:00.973285 23543 consensus_queue.cc:1048] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
I20251023 02:05:01.013602 23675 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:01.014074 23585 raft_consensus.cc:3055] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:01.014463 23585 raft_consensus.cc:740] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:01.014668 23679 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251023 02:05:01.015255 23585 consensus_queue.cc:260] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.015698 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:01.016398 23585 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:01.018273 23735 consensus_queue.cc:1048] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:05:01.008064 22035 raft_consensus.cc:1240] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:00.946493 23297 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:01.028848 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 48928.9us
I20251023 02:05:01.011716 23541 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:00.993422 23709 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:00.995188 23704 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:01.045418 22037 raft_consensus.cc:1240] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:01.047659 22077 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:01.055722 22076 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:01.056588 22077 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:00.991815 23473 consensus_queue.cc:1059] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20251023 02:05:01.058290 22076 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:01.040827 23491 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:01.042383 23717 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:01.054430 23704 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:01.048869 23558 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:01.042934 23727 raft_consensus.cc:697] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:01.039618 23630 consensus_queue.cc:1048] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20251023 02:05:01.046626 23467 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.091936 23710 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:01.100737 23710 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:01.061769 23637 consensus_queue.cc:1059] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20251023 02:05:01.101107 23710 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:01.067113 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:01.043133 23740 consensus_queue.cc:1048] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20251023 02:05:01.069979 23491 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.067133 23642 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:01.102599 22105 raft_consensus.cc:1240] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251023 02:05:01.054235 23716 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:01.106962 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:01.101503 23541 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:01.061990 23703 consensus_queue.cc:1059] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20251023 02:05:01.114364 22102 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:01.115743 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.069242 23678 consensus_queue.cc:1059] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.047s
I20251023 02:05:01.117198 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:01.119047 22000 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:01.122553 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 140566us
I20251023 02:05:01.119881 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.125818 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.071499 23464 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20251023 02:05:01.131165 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:01.068807 23738 consensus_queue.cc:1048] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.065s
I20251023 02:05:01.104332 23709 consensus_queue.cc:1059] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
W20251023 02:05:01.123435 23674 consensus_meta.cc:287] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.567s user 0.008s sys 0.003s
W20251023 02:05:01.148964 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:01.148921 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:01.121930 23642 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:01.122952 21858 catalog_manager.cc:5649] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 3, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:01.176280 22108 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:01.179769 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.179276 21858 catalog_manager.cc:5649] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 2, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 2 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:05:01.119231 23491 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:01.117524 23735 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:01.120783 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.180783 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.171343 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:01.132254 23671 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:01.130817 23541 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:01.189939 22002 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:01.172986 23738 consensus_queue.cc:1243] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251023 02:05:01.141602 23467 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:01.192668 21856 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56856 (request call id 227) took 9924 ms. Trace:
I20251023 02:05:01.192822 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.201611 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.202337 22108 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:01.183882 23669 consensus_meta.cc:287] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.732s user 0.012s sys 0.000s
I20251023 02:05:01.197443 22002 leader_election.cc:400] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:05:01.206904 22002 leader_election.cc:403] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251023 02:05:01.208636 22002 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:01.210898 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 160337us
I20251023 02:05:01.183501 21858 catalog_manager.cc:5649] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 2 to 4, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4), now has a pending config: VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1), VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2), VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3), VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 4 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:01.185294 23743 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:01.211261 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:01.191727 23520 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20251023 02:05:01.178686 23674 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.193109 23735 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:00.934393 22104 rpcz_store.cc:276] 1023 02:04:58.932662 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.013920 (+1081258us) service_pool.cc:225] Handling call
1023 02:05:00.055953 (+ 42033us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:00.067088 (+ 11135us) raft_consensus.cc:1534] Early marking committed up to index 5
1023 02:05:00.387572 (+320484us) log.cc:844] Serialized 14 byte log entry
1023 02:05:00.825321 (+437749us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:00.827537 (+ 2216us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:00.841384 (+ 13847us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:00.853250 (+ 11866us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:00.862007 (+ 8757us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":202972,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":192551,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":75,"thread_start_us":641906,"threads_started":3,"wal-append.queue_time_us":263043}
I20251023 02:05:01.206209 23671 raft_consensus.cc:697] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:01.210661 23689 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:05:01.210621 23407 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:05:01.212293 23669 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.217250 21858 catalog_manager.cc:5649] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 2, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 2 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:05:01.200667 21856 rpcz_store.cc:276] 1023 02:04:51.255474 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:51.255679 (+ 205us) service_pool.cc:225] Handling call
1023 02:05:01.179883 (+9924204us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:04:58.903829 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:04:59.227061 (+323232us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:04:59.230940 (+ 3879us) write_op.cc:435] Acquired schema lock
1023 02:04:59.231849 (+ 909us) tablet.cc:660] Decoding operations
1023 02:04:59.336526 (+104677us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:04:59.338261 (+ 1735us) write_op.cc:641] Partition lock acquired for write op
1023 02:04:59.340577 (+ 2316us) tablet.cc:683] Acquiring locks for 3 operations
1023 02:04:59.371287 (+ 30710us) tablet.cc:699] Row locks acquired
1023 02:04:59.371308 (+ 21us) write_op.cc:260] PREPARE: finished
1023 02:04:59.371452 (+ 144us) write_op.cc:270] Start()
1023 02:04:59.371566 (+ 114us) write_op.cc:276] Timestamp: P: 1761185099371410 usec, L: 0
1023 02:04:59.371588 (+ 22us) op_driver.cc:348] REPLICATION: starting
1023 02:04:59.372167 (+ 579us) log.cc:844] Serialized 1298 byte log entry
1023 02:04:59.491650 (+119483us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5c9cdfdc 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c5f62f2bb
1023 02:05:00.212594 (+720944us) op_driver.cc:464] REPLICATION: finished
1023 02:05:00.303442 (+ 90848us) write_op.cc:301] APPLY: starting
1023 02:05:00.305042 (+ 1600us) tablet.cc:1365] starting BulkCheckPresence
1023 02:05:00.331580 (+ 26538us) tablet.cc:1368] finished BulkCheckPresence
1023 02:05:00.336514 (+ 4934us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:05:00.474721 (+138207us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:05:00.474915 (+ 194us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1023 02:05:00.474972 (+ 57us) write_op.cc:312] APPLY: finished
1023 02:05:00.475191 (+ 219us) log.cc:844] Serialized 34 byte log entry
1023 02:05:00.475356 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:05:00.475463 (+ 107us) write_op.cc:454] Released schema lock
1023 02:05:00.475507 (+ 44us) write_op.cc:341] FINISH: Updating metrics
1023 02:05:00.475849 (+ 342us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:05:00.475876 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":90588,"apply.run_cpu_time_us":2460,"apply.run_wall_time_us":172385,"num_ops":3,"prepare.queue_time_us":135027,"prepare.run_cpu_time_us":28307,"prepare.run_wall_time_us":860150,"raft.queue_time_us":254304,"raft.run_cpu_time_us":31991,"raft.run_wall_time_us":78506,"replication_time_us":837106,"spinlock_wait_cycles":10121600,"thread_start_us":656103,"threads_started":4,"wal-append.queue_time_us":536819}]]}
W20251023 02:05:01.284379 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 227907us
I20251023 02:05:01.219221 22002 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:01.183130 23739 consensus_queue.cc:1059] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20251023 02:05:01.239013 23547 raft_consensus.cc:3055] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:01.240216 23738 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:01.246881 23713 raft_consensus.cc:3055] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:01.217762 23297 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:01.256296 23720 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:01.258046 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.312585 22111 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:01.260200 22108 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:01.314229 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:01.315595 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.316233 22108 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:01.199196 23727 consensus_queue.cc:237] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.203716 23751 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:01.194314 23541 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.276793 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:01.222694 22102 raft_consensus.cc:2393] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
W20251023 02:05:01.265041 23721 consensus_meta.cc:287] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.628s user 0.000s sys 0.009s
W20251023 02:05:01.286891 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 67764.2us
I20251023 02:05:01.326378 22002 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:01.261570 23646 raft_consensus.cc:3055] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:05:01.287460 22109 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:01.290125 23689 raft_consensus.cc:2749] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:05:01.284698 21858 catalog_manager.cc:5649] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:05:01.319460 22037 raft_consensus.cc:2393] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:01.319415 22000 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:01.321094 21925 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:01.339308 21925 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:01.339339 22002 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:01.340610 21925 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:01.289340 23547 raft_consensus.cc:740] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:01.263144 23456 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:01.326841 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:01.287662 22077 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:01.290177 23729 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:01.367314 21858 catalog_manager.cc:5649] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 4, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 4 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:05:01.398059 21858 catalog_manager.cc:5649] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 2, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 2 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:05:01.292616 23713 raft_consensus.cc:740] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:01.374323 21925 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:01.409309 21925 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:01.387163 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 47880.8us
I20251023 02:05:01.318385 23716 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:01.392864 21958 raft_consensus.cc:2368] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:05:01.412632 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:01.413390 21958 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
W20251023 02:05:01.394475 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 106889us
I20251023 02:05:01.343899 22111 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:01.418993 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1310) took 1101 ms. Trace:
I20251023 02:05:01.419435 22111 rpcz_store.cc:276] 1023 02:05:00.317302 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.330416 (+ 13114us) service_pool.cc:225] Handling call
1023 02:05:01.257862 (+927446us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:05:01.418923 (+161061us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8419712}
I20251023 02:05:01.421175 22000 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:01.425320 22002 leader_election.cc:400] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:05:01.425880 22002 leader_election.cc:403] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251023 02:05:01.327698 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.327625 23646 raft_consensus.cc:740] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:01.466650 22622 raft_consensus.cc:1240] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:05:01.418149 22077 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:01.431236 21858 catalog_manager.cc:5649] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 3, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 3 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } }
W20251023 02:05:01.464927 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 43864.8us
I20251023 02:05:01.313278 23738 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.395622 23731 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:01.394239 23637 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:01.336182 23721 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.390960 23751 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
W20251023 02:05:01.473618 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 48493.3us
I20251023 02:05:01.471486 21858 catalog_manager.cc:5649] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 2, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 2 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251023 02:05:01.547014 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 128966us
I20251023 02:05:01.499729 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:01.394081 23737 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:01.516625 23735 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:01.519534 22587 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:01.397991 23757 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:01.463336 23756 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:01.547993 22077 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:01.431458 23297 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:01.463897 23729 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:01.336506 23720 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.392347 23750 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:01.517225 23739 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:01.473694 23691 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:01.583961 22107 raft_consensus.cc:2368] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4.
I20251023 02:05:01.584527 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:01.593267 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 45396.1us
I20251023 02:05:01.594821 22077 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:01.540261 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:01.517757 23754 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251023 02:05:01.560001 21955 consensus_meta.cc:287] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.703s user 0.008s sys 0.004s
W20251023 02:05:01.587666 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 375097us
I20251023 02:05:01.598270 22077 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:01.663221 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 65048.1us
I20251023 02:05:01.664215 22077 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:01.547740 21858 catalog_manager.cc:5649] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 3, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4), now has a pending config: VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1), VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2), VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3), VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 3 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:01.667024 22077 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:01.669305 23749 consensus_queue.cc:1059] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251023 02:05:01.670540 23753 raft_consensus.cc:3055] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:05:01.670981 23753 raft_consensus.cc:740] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:01.674332 21958 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:01.577549 23717 raft_consensus.cc:697] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:01.394780 22108 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:01.580271 23657 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:01.691581 22032 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:01.583009 23735 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251023 02:05:01.710155 23759 log.cc:927] Time spent T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20251023 02:05:01.710600 23735 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:01.672916 23691 raft_consensus.cc:697] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:01.714962 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:01.582871 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:01.641988 23642 raft_consensus.cc:697] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:01.708739 22000 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:01.638376 23637 raft_consensus.cc:2764] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251023 02:05:01.712971 23691 consensus_queue.cc:237] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.534516 23547 consensus_queue.cc:260] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.587527 23760 raft_consensus.cc:2749] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:01.693303 23674 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:01.577426 23737 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:01.747406 23585 consensus_meta.cc:287] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.731s user 0.010s sys 0.000s
I20251023 02:05:01.748528 23585 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:05:01.748736 23737 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:01.672405 21955 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:01.740057 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1246) took 3071 ms. Trace:
I20251023 02:05:01.740417 23541 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:01.741673 23547 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:01.731096 23753 consensus_queue.cc:260] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:01.782058 23456 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:01.769188 21953 rpcz_store.cc:276] 1023 02:04:58.668197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:59.943169 (+1274972us) service_pool.cc:225] Handling call
1023 02:04:59.943595 (+ 426us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:00.533907 (+590312us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:01.730428 (+1196521us) spinlock_profiling.cc:243] Waited 526 ms on lock 0x7b54000bead4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:01.739987 (+ 9559us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1209235200}
I20251023 02:05:01.578780 23713 consensus_queue.cc:260] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.746860 23738 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:01.701448 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:01.589645 23729 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:01.534389 23671 consensus_queue.cc:237] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.719147 23716 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:01.756940 21957 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:01.598824 23297 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:01.772722 23750 raft_consensus.cc:697] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:01.808851 23766 log.cc:927] Time spent T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20251023 02:05:01.809038 23750 consensus_queue.cc:237] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:01.788527 23456 consensus_queue.cc:1059] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.028s
I20251023 02:05:01.818457 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1282) took 1152 ms. Trace:
I20251023 02:05:01.818612 21956 raft_consensus.cc:1240] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:01.818851 21961 rpcz_store.cc:276] 1023 02:05:00.666352 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.686398 (+ 20046us) service_pool.cc:225] Handling call
1023 02:05:01.818270 (+1131872us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54000bf4d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:01.818409 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":262580480}
I20251023 02:05:01.773206 21958 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:01.725936 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:01.776118 23737 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:01.776826 23751 raft_consensus.cc:697] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:01.779394 23674 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:01.780066 23747 log.cc:927] Time spent T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20251023 02:05:01.780395 23771 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:01.780493 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:01.583451 23757 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:01.587003 23756 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:01.779183 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 70550.5us
I20251023 02:05:01.727511 23669 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:01.599014 23646 consensus_queue.cc:260] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:01.786758 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:01.726384 23762 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:05:01.770546 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 361338us
I20251023 02:05:01.891119 21925 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:01.787192 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 120265us
I20251023 02:05:01.840078 23737 raft_consensus.cc:697] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:01.731549 23765 raft_consensus.cc:2749] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:01.749677 23761 raft_consensus.cc:2749] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:01.757499 23585 raft_consensus.cc:2749] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251023 02:05:01.797971 23745 log.cc:927] Time spent T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.693s user 0.008s sys 0.005s
I20251023 02:05:01.802091 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1000) took 1507 ms. Trace:
I20251023 02:05:01.802753 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1299) took 1502 ms. Trace:
I20251023 02:05:01.803537 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1303) took 1498 ms. Trace:
I20251023 02:05:01.821318 22002 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:01.783491 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1276) took 1119 ms. Trace:
I20251023 02:05:01.822209 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:01.819986 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:01.939272 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 118055us
I20251023 02:05:01.939623 21961 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:01.800228 23731 raft_consensus.cc:697] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:01.803962 22104 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:05:01.892099 22077 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:01.945080 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 53070.7us
W20251023 02:05:01.791056 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:01.947638 22036 raft_consensus.cc:1240] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:01.844473 23760 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:01.950577 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1283) took 1284 ms. Trace:
I20251023 02:05:01.950973 21954 rpcz_store.cc:276] 1023 02:05:00.666642 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.840738 (+174096us) service_pool.cc:225] Handling call
1023 02:05:00.841127 (+ 389us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:00.877975 (+ 36848us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:00.878019 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:00.928079 (+ 50060us) log.cc:844] Serialized 26 byte log entry
1023 02:05:01.278196 (+350117us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:05:01.278921 (+ 725us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:01.279001 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:01.950369 (+671368us) raft_consensus.cc:1685] finished
1023 02:05:01.950396 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:01.950518 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9529,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":17437,"thread_start_us":322156,"threads_started":1,"wal-append.queue_time_us":635517}
W20251023 02:05:01.893043 23767 log.cc:927] Time spent T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20251023 02:05:01.824579 23774 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:01.831635 23751 consensus_queue.cc:237] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:01.934078 22103 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1305) took 1618 ms (client timeout 2072 ms). Trace:
I20251023 02:05:01.935381 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 111) took 1268 ms. Trace:
I20251023 02:05:01.887120 23721 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:01.938616 22110 rpcz_store.cc:276] 1023 02:05:00.305335 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.305475 (+ 140us) service_pool.cc:225] Handling call
1023 02:05:00.334092 (+ 28617us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:00.481552 (+147460us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:01.803320 (+1321768us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b5400021554. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:01.803486 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":337860608}
I20251023 02:05:01.938371 22034 rpcz_store.cc:276] 1023 02:05:00.294498 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.298632 (+ 4134us) service_pool.cc:225] Handling call
1023 02:05:01.801925 (+1503293us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b54002a8fd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:01.802038 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":744616704}
I20251023 02:05:01.930156 21953 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:01.890933 23754 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:01.938489 22109 rpcz_store.cc:276] 1023 02:05:00.300555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.301429 (+ 874us) service_pool.cc:225] Handling call
1023 02:05:00.323267 (+ 21838us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:01.327891 (+1004624us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:01.802522 (+474631us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b54000aaad4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:01.802697 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2217275136}
I20251023 02:05:01.939069 22108 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:02.003376 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1333) took 1001 ms. Trace:
W20251023 02:05:01.938422 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 224393us
I20251023 02:05:01.959291 21962 rpcz_store.cc:276] 1023 02:05:00.666868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.943770 (+276902us) service_pool.cc:225] Handling call
1023 02:05:00.956399 (+ 12629us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:00.956739 (+ 340us) raft_consensus.cc:1534] Early marking committed up to index 5
1023 02:05:01.107643 (+150904us) log.cc:844] Serialized 14 byte log entry
1023 02:05:01.935032 (+827389us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:01.935090 (+ 58us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:01.935136 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:01.935202 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:01.935330 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16643,"prepare.queue_time_us":258216,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":228058,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":7804,"thread_start_us":932345,"threads_started":3,"wal-append.queue_time_us":593542}
I20251023 02:05:02.003798 22108 rpcz_store.cc:276] 1023 02:05:01.001661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:01.821955 (+820294us) service_pool.cc:225] Handling call
1023 02:05:02.003306 (+181351us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:02.004707 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:01.840492 23642 consensus_queue.cc:237] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.792905 23717 consensus_queue.cc:237] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:01.831717 23780 consensus_queue.cc:1059] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
I20251023 02:05:01.960702 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:01.930864 21925 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:01.961164 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:01.936405 23773 raft_consensus.cc:2749] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20251023 02:05:02.026881 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 132674us
I20251023 02:05:01.959710 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:02.027426 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:01.935122 23772 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:02.030977 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:02.032747 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:02.033021 21961 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:01.840633 23674 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.005867 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:02.033926 21954 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:02.006578 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:02.034734 22002 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:02.009940 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:01.955346 23779 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:02.010002 23731 consensus_queue.cc:237] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:02.022924 23733 consensus_meta.cc:287] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.146s user 0.011s sys 0.000s
I20251023 02:05:01.955818 23760 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.016630 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1280) took 1350 ms. Trace:
I20251023 02:05:02.036794 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:02.057909 21959 rpcz_store.cc:276] 1023 02:05:00.665894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.815645 (+149751us) service_pool.cc:225] Handling call
1023 02:05:00.816098 (+ 453us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:00.832987 (+ 16889us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:00.833013 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:00.844645 (+ 11632us) log.cc:844] Serialized 52 byte log entry
1023 02:05:00.951377 (+106732us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:01.184270 (+232893us) log.cc:844] Serialized 14 byte log entry
1023 02:05:01.532685 (+348415us) log.cc:844] Serialized 14 byte log entry
1023 02:05:01.722388 (+189703us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:01.722480 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:01.797127 (+ 74647us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:02.016436 (+219309us) raft_consensus.cc:1685] finished
1023 02:05:02.016468 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.016583 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":729,"prepare.queue_time_us":234419,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":4175,"raft.queue_time_us":390022,"raft.run_cpu_time_us":452,"raft.run_wall_time_us":25732,"spinlock_wait_cycles":4436736,"thread_start_us":762632,"threads_started":4,"wal-append.queue_time_us":179253}
I20251023 02:05:02.059966 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.060127 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:02.060161 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:02.005950 23585 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:02.017863 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:02.020749 23710 consensus_meta.cc:287] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.919s user 0.005s sys 0.003s
I20251023 02:05:01.941917 21955 rpcz_store.cc:276] 1023 02:05:00.656222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.659378 (+ 3156us) service_pool.cc:225] Handling call
1023 02:05:01.775833 (+1116455us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:01.958087 23765 consensus_queue.cc:1059] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.064s
I20251023 02:05:02.022878 23757 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:01.961078 23777 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:01.865183 23771 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.035346 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:02.039233 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.076427 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:02.040405 22077 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:02.041698 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:02.053821 23725 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:02.024453 23727 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:02.043175 23756 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:02.074680 23558 consensus_meta.cc:287] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.964s user 0.004s sys 0.006s
I20251023 02:05:02.044059 23780 consensus_queue.cc:1059] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:01.957441 23774 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.044754 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.044864 23739 raft_consensus.cc:2749] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:05:02.024699 23721 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:02.036128 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.002349 23754 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.038103 23761 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:02.045894 23779 raft_consensus.cc:3055] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:05:02.045933 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1313) took 1670 ms. Trace:
I20251023 02:05:02.046556 22028 raft_consensus.cc:1240] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20251023 02:05:01.933909 23763 consensus_queue.cc:1175] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "818907d63f1f4d84ba267ebe6ee837e0" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251023 02:05:02.048028 22035 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:02.056550 23729 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.056746 23776 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:02.046938 23746 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.843s user 0.005s sys 0.026s
I20251023 02:05:02.126281 23790 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:02.059188 23760 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.062731 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.316283 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1255) took 1028 ms. Trace:
I20251023 02:05:02.316751 22110 rpcz_store.cc:276] 1023 02:05:01.288012 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.027957 (+739945us) service_pool.cc:225] Handling call
1023 02:05:02.316209 (+288252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:01.954285 23720 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:02.018003 23704 consensus_meta.cc:287] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.907s user 0.010s sys 0.000s
I20251023 02:05:02.060830 21959 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:02.071060 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.297722 23779 raft_consensus.cc:740] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:02.029888 23637 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:02.029697 23716 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:02.073746 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.053599 23541 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:05:02.041602 23792 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:02.002362 23737 consensus_queue.cc:237] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.081156 23771 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.054455 23773 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:02.085215 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.091538 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.325006 23704 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.099006 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.339138 23774 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.339224 23754 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.070680 23585 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.049880 23772 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.339977 23704 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:02.105940 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 65621.9us
I20251023 02:05:02.105979 22002 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:02.043823 23674 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:02.109898 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 110) took 1443 ms. Trace:
I20251023 02:05:02.113183 23731 consensus_queue.cc:1059] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:05:02.122252 23795 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:02.124524 22109 raft_consensus.cc:2410] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251023 02:05:02.294926 21954 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
W20251023 02:05:02.152263 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:02.297806 22106 rpcz_store.cc:276] 1023 02:05:00.375174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.461947 (+ 86773us) service_pool.cc:225] Handling call
1023 02:05:00.462216 (+ 269us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:01.259933 (+797717us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b1800049780. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5f63d55c 00007f9c5c9ce3bb 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9e8445 00007f9c5fa90c00 00007f9c5fa9179b 00007f9c5fa9c1db
1023 02:05:01.260092 (+ 159us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:01.260131 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:01.260683 (+ 552us) log.cc:844] Serialized 26 byte log entry
1023 02:05:01.533662 (+272979us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:01.533702 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:01.533767 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:02.045658 (+511891us) raft_consensus.cc:1685] finished
1023 02:05:02.045679 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.045879 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":69577,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":4913,"spinlock_wait_cycles":9139072,"thread_start_us":347865,"threads_started":2,"wal-append.queue_time_us":477426}
I20251023 02:05:02.343624 23777 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:02.351783 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1015) took 1287 ms. Trace:
I20251023 02:05:02.336189 23541 raft_consensus.cc:697] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:02.124027 23762 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:02.352304 21954 rpcz_store.cc:276] 1023 02:05:01.064300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:01.955402 (+891102us) service_pool.cc:225] Handling call
1023 02:05:02.351720 (+396318us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.301821 21961 raft_consensus.cc:2410] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251023 02:05:02.072927 23710 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.353634 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:02.353511 23779 consensus_queue.cc:260] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.354622 23779 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:02.355006 23710 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:02.066344 23733 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.103159 23725 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.124599 23756 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:02.126099 23558 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:02.127957 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:02.297537 22618 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:05:02.059144 23297 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.298770 23456 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:02.364635 23456 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.319276 21925 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:02.322085 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:02.322980 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:01.958849 22103 rpcz_store.cc:269] 1023 02:05:00.313972 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.314463 (+ 491us) service_pool.cc:225] Handling call
1023 02:05:01.768998 (+1454535us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:05:01.932198 (+163200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8726400}
I20251023 02:05:02.366519 23456 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:02.367195 22103 raft_consensus.cc:1240] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:02.367463 21955 raft_consensus.cc:2393] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:02.367849 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1314) took 1983 ms. Trace:
I20251023 02:05:02.368194 22103 rpcz_store.cc:276] 1023 02:05:00.384450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.366687 (+1982237us) service_pool.cc:225] Handling call
1023 02:05:02.367015 (+ 328us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.367601 (+ 586us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.367794 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.323760 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1304) took 1194 ms. Trace:
I20251023 02:05:02.369343 21959 rpcz_store.cc:276] 1023 02:05:01.129719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.059907 (+930188us) service_pool.cc:225] Handling call
1023 02:05:02.323705 (+263798us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.370293 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:02.370800 21959 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:02.373730 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.373970 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.115744 23727 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.374881 23725 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:02.375787 22109 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:02.376317 23727 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.309037 23761 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:02.377256 21953 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:02.329885 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.323271 23485 raft_consensus.cc:3055] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:02.378080 22110 raft_consensus.cc:2393] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:02.378134 23485 raft_consensus.cc:740] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:02.379017 23485 consensus_queue.cc:260] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.330868 21926 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:02.326921 23720 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:02.324996 23760 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:02.385995 23760 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:02.341226 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 235339us
I20251023 02:05:02.387943 21959 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:05:02.341943 21962 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:05:02.367849 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1290) took 1242 ms. Trace:
I20251023 02:05:02.388588 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.388764 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.389550 22029 raft_consensus.cc:2410] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:05:02.341943 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1309) took 2025 ms. Trace:
I20251023 02:05:02.343113 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.390291 23760 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.344362 21957 rpcz_store.cc:276] 1023 02:05:00.666592 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.834346 (+167754us) service_pool.cc:225] Handling call
1023 02:05:01.756744 (+922398us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:01.801519 (+ 44775us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.109658 (+308139us) spinlock_profiling.cc:243] Waited 922 ms on lock 0x7b540030f7d0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:02.109824 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2120894720}
I20251023 02:05:02.388772 21955 rpcz_store.cc:276] 1023 02:05:01.125551 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.077005 (+951454us) service_pool.cc:225] Handling call
1023 02:05:02.367801 (+290796us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.390863 23760 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:05:02.391307 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:02.391418 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:02.362803 23772 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.392030 21957 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:02.392498 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:02.352308 23777 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.392776 21957 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7"
dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:40860
I20251023 02:05:02.343456 23773 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.310426 23397 consensus_queue.cc:1059] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:02.356381 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.394266 22030 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:02.395023 21954 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:02.394810 23720 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.395457 23773 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.352550 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.396265 22106 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:02.397964 22106 raft_consensus.cc:1275] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:02.398633 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1319) took 1918 ms. Trace:
I20251023 02:05:02.398905 22106 rpcz_store.cc:276] 1023 02:05:00.480185 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.397173 (+1916988us) service_pool.cc:225] Handling call
1023 02:05:02.397548 (+ 375us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.398364 (+ 816us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.398586 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.400182 22106 raft_consensus.cc:1240] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:05:02.400890 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1320) took 1920 ms. Trace:
I20251023 02:05:02.318281 23776 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.403986 23720 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.404449 23776 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.336589 23792 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.358301 23733 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:02.352934 23541 consensus_queue.cc:237] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.360106 23795 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:02.356963 23762 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:02.330498 23716 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.343906 23729 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:02.342038 23771 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:02.416350 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.417091 22111 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:02.323012 23790 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.360113 23756 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.418437 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.352655 23799 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:02.363533 22617 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20251023 02:05:02.376958 23761 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:02.336346 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 296357us
I20251023 02:05:02.423501 22000 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:05:02.382599 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 63448us
W20251023 02:05:02.383466 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 52737.9us
I20251023 02:05:02.425773 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1016) took 1361 ms. Trace:
I20251023 02:05:02.384225 22075 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:02.428056 22589 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:02.360648 22102 raft_consensus.cc:2393] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:02.385257 23297 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:02.385457 23754 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:02.136819 23721 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.390030 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.393262 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1291) took 1262 ms. Trace:
I20251023 02:05:02.393324 21957 raft_consensus.cc:606] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:02.434204 21957 raft_consensus.cc:614] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:02.401269 22106 rpcz_store.cc:276] 1023 02:05:00.480453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.399638 (+1919185us) service_pool.cc:225] Handling call
1023 02:05:02.399954 (+ 316us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.400622 (+ 668us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.400827 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.434896 21926 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:02.407742 22105 rpcz_store.cc:276] 1023 02:05:00.317025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.337663 (+ 20638us) service_pool.cc:225] Handling call
1023 02:05:01.092266 (+754603us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:05:01.099554 (+ 7288us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:01.106080 (+ 6526us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.341708 (+1235628us) spinlock_profiling.cc:243] Waited 2.78 ms on lock 0x7b540032fad4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:02.341888 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11449472}
I20251023 02:05:02.434098 21953 rpcz_store.cc:276] 1023 02:05:01.125868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:01.784516 (+658648us) service_pool.cc:225] Handling call
1023 02:05:02.388379 (+603863us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.443095 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.444229 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.445192 22105 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:02.445982 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.446672 22105 raft_consensus.cc:2393] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:02.408944 21954 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:05:02.409456 21959 raft_consensus.cc:1240] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:05:02.447423 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1284) took 1780 ms. Trace:
I20251023 02:05:02.447758 21959 rpcz_store.cc:276] 1023 02:05:00.666868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.408875 (+1742007us) service_pool.cc:225] Handling call
1023 02:05:02.409247 (+ 372us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.447160 (+ 37913us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.447358 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.448836 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:02.449198 21959 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:02.409508 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.410671 23743 raft_consensus.cc:3055] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:05:02.410936 23792 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.411203 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.416265 23716 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:02.416846 23771 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.454218 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:02.417171 23795 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:02.417371 23762 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:02.362836 23585 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.327793 23637 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.420603 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1256) took 1132 ms. Trace:
I20251023 02:05:02.419467 23756 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.448331 23720 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:05:02.426141 21958 rpcz_store.cc:276] 1023 02:05:01.064610 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:01.549146 (+484536us) service_pool.cc:225] Handling call
1023 02:05:02.425613 (+876467us) spinlock_profiling.cc:243] Waited 81.6 ms on lock 0x7b540017b954. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:02.425718 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":187666688}
I20251023 02:05:02.428535 21925 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:02.421355 22617 pending_rounds.cc:85] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting all ops after (but not including) 3
I20251023 02:05:02.461436 21925 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:02.430162 22002 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:02.430917 23763 consensus_queue.cc:1059] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.025s
I20251023 02:05:02.408030 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1317) took 1928 ms. Trace:
I20251023 02:05:02.394474 23777 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.438119 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40860 (request call id 7) took 1434 ms. Trace:
I20251023 02:05:02.438176 23801 consensus_queue.cc:1059] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:05:02.438406 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:02.465859 21957 rpcz_store.cc:276] 1023 02:05:01.000361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.392668 (+1392307us) service_pool.cc:225] Handling call
1023 02:05:02.434605 (+ 41937us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.464457 22110 rpcz_store.cc:276] 1023 02:05:00.479580 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.378741 (+1899161us) service_pool.cc:225] Handling call
1023 02:05:02.379034 (+ 293us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.379218 (+ 184us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:05:02.379510 (+ 292us) log.cc:844] Serialized 14 byte log entry
1023 02:05:02.407577 (+ 28067us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.407662 (+ 85us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:02.407724 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.407815 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.407966 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2392,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":94,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":2212,"thread_start_us":27155,"threads_started":1,"wal-append.queue_time_us":26851}
I20251023 02:05:02.439182 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.440800 23646 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:02.441825 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.450974 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:02.451699 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.431398 23776 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:02.444200 23729 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:02.474225 22105 raft_consensus.cc:2393] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:02.474498 23729 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:02.474748 22622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 245) took 1541 ms. Trace:
I20251023 02:05:02.474956 23729 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:02.475055 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.456678 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:02.475726 23491 raft_consensus.cc:3055] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:02.451361 23726 consensus_queue.cc:1048] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20251023 02:05:02.476126 23491 raft_consensus.cc:740] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:02.476244 23771 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.457563 22111 rpcz_store.cc:276] 1023 02:05:01.288320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.416110 (+1127790us) service_pool.cc:225] Handling call
1023 02:05:02.420559 (+ 4449us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.470923 23690 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:05:02.477016 23726 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:02.476902 23491 consensus_queue.cc:260] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.477449 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.475776 22105 raft_consensus.cc:2393] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 2: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:02.478335 21957 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:02.478462 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.478940 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 112) took 1811 ms. Trace:
I20251023 02:05:02.422138 21928 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:02.459235 22033 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:02.450728 23743 raft_consensus.cc:740] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:02.455020 23773 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:02.467046 23756 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:02.485706 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.486299 22111 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:02.487249 21959 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:02.487309 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.488095 22111 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:02.489059 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.463136 22002 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:02.435644 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1323) took 1953 ms. Trace:
I20251023 02:05:02.463618 23792 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:02.463754 23585 raft_consensus.cc:2749] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:02.430323 23799 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.458887 23795 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:02.499228 23795 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.501276 23795 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.468859 22077 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:02.461601 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.462330 22617 pending_rounds.cc:107] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.4
I20251023 02:05:02.473701 23721 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.504385 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.467291 23541 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:02.445936 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:02.443508 23725 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:02.479256 21957 rpcz_store.cc:276] 1023 02:05:00.667121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.466948 (+1799827us) service_pool.cc:225] Handling call
1023 02:05:02.478188 (+ 11240us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.478737 (+ 549us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.478896 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.475102 22622 rpcz_store.cc:276] 1023 02:05:00.933469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.933641 (+ 172us) service_pool.cc:225] Handling call
1023 02:05:00.933937 (+ 296us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:01.469919 (+535982us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.474499 (+1004580us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b5400181854. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:02.474694 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1224784256}
I20251023 02:05:02.482023 23491 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:02.482470 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.482625 23761 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.482739 23771 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251023 02:05:02.479542 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 57465.4us
I20251023 02:05:02.514688 22617 raft_consensus.cc:2917] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Aborting config change with OpId 3.4: Aborted: Op aborted by new leader
I20251023 02:05:02.523092 23541 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:02.466768 23637 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:02.443766 23297 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.482470 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.492417 22106 rpcz_store.cc:276] 1023 02:05:00.482469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.434913 (+1952444us) service_pool.cc:225] Handling call
1023 02:05:02.435195 (+ 282us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.435353 (+ 158us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.435376 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.435424 (+ 48us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:02.435450 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.435501 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.435594 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.559867 22106 raft_consensus.cc:1240] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:02.525790 23725 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:02.498343 23792 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.551105 23771 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:02.564244 23771 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:02.503031 23790 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.477499 23690 raft_consensus.cc:697] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:02.468297 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.567461 22110 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:02.568162 23818 consensus_queue.cc:1059] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:05:02.568892 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 141) took 1911 ms. Trace:
I20251023 02:05:02.569169 22110 rpcz_store.cc:276] 1023 02:05:00.657334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.568081 (+1910747us) service_pool.cc:225] Handling call
1023 02:05:02.568410 (+ 329us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.568587 (+ 177us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.568614 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.568670 (+ 56us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:02.568698 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.568748 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.568842 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.512382 22077 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:02.473603 23776 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:02.570195 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 57911.8us
I20251023 02:05:02.570386 22110 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251023 02:05:02.570895 22077 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:02.571082 23777 raft_consensus.cc:2749] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:02.571636 23776 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.490633 23756 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:02.572253 23756 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:02.571642 23777 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:02.570920 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 143) took 1913 ms. Trace:
I20251023 02:05:02.573598 22110 rpcz_store.cc:276] 1023 02:05:00.657999 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.569817 (+1911818us) service_pool.cc:225] Handling call
1023 02:05:02.570084 (+ 267us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.570716 (+ 632us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.570876 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.574457 22110 raft_consensus.cc:1240] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:02.574841 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1243) took 1913 ms. Trace:
I20251023 02:05:02.575103 22110 rpcz_store.cc:276] 1023 02:05:00.661174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.574115 (+1912941us) service_pool.cc:225] Handling call
1023 02:05:02.574337 (+ 222us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.574703 (+ 366us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.574808 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.576000 22110 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:02.576408 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1247) took 1913 ms. Trace:
I20251023 02:05:02.576613 22110 rpcz_store.cc:276] 1023 02:05:00.663424 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.575597 (+1912173us) service_pool.cc:225] Handling call
1023 02:05:02.575817 (+ 220us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.576265 (+ 448us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.576376 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.577447 22110 raft_consensus.cc:1240] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:02.518774 23768 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:02.521785 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.512897 23737 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:02.543325 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1324) took 1895 ms. Trace:
I20251023 02:05:02.563520 23725 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:02.484364 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:02.514644 23801 consensus_queue.cc:1059] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20251023 02:05:02.491813 23773 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.560299 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1325) took 1911 ms. Trace:
W20251023 02:05:02.495833 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:02.522329 23803 consensus_queue.cc:1059] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251023 02:05:02.502386 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.503997 23743 consensus_queue.cc:260] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.555982 23541 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:02.483310 23726 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:02.564252 23811 log.cc:927] Time spent T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20251023 02:05:02.578965 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1251) took 1914 ms. Trace:
I20251023 02:05:02.560823 22617 raft_consensus.cc:2889] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:02.547470 23729 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.579530 22111 rpcz_store.cc:276] 1023 02:05:00.648550 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.492655 (+1844105us) service_pool.cc:225] Handling call
1023 02:05:02.492987 (+ 332us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.493230 (+ 243us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:05:02.494130 (+ 900us) log.cc:844] Serialized 14 byte log entry
1023 02:05:02.542979 (+ 48849us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.543033 (+ 54us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:02.543077 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.543140 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.543266 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":723,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":980,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":101,"thread_start_us":47235,"threads_started":1,"wal-append.queue_time_us":58240}
W20251023 02:05:02.586359 23806 log.cc:927] Time spent T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20251023 02:05:02.582990 22106 rpcz_store.cc:276] 1023 02:05:00.648949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.559402 (+1910453us) service_pool.cc:225] Handling call
1023 02:05:02.559682 (+ 280us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.560145 (+ 463us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.560260 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.593101 22111 raft_consensus.cc:1275] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:02.593222 22106 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20251023 02:05:02.595523 22106 pending_rounds.cc:85] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 3
I20251023 02:05:02.596062 22106 pending_rounds.cc:107] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.4
I20251023 02:05:02.597352 22106 raft_consensus.cc:2917] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Aborting config change with OpId 3.4: Aborted: Op aborted by new leader
I20251023 02:05:02.598148 22106 raft_consensus.cc:2889] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:02.581979 22105 raft_consensus.cc:1240] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:02.598879 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 144) took 1919 ms. Trace:
I20251023 02:05:02.599195 22105 rpcz_store.cc:276] 1023 02:05:00.679866 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.581532 (+1901666us) service_pool.cc:225] Handling call
1023 02:05:02.581820 (+ 288us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.598635 (+ 16815us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.598828 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.600044 23729 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.597977 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1329) took 1736 ms. Trace:
I20251023 02:05:02.601538 22111 rpcz_store.cc:276] 1023 02:05:00.857987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.592323 (+1734336us) service_pool.cc:225] Handling call
1023 02:05:02.592704 (+ 381us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.593543 (+ 839us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.594325 (+ 782us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.602463 22111 raft_consensus.cc:1240] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:05:02.603798 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1335) took 1571 ms. Trace:
I20251023 02:05:02.596038 23792 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.634621 23756 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.635416 23773 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:02.634212 23809 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20251023 02:05:02.641278 23756 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:02.641531 23812 log.cc:927] Time spent T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20251023 02:05:02.644749 23771 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:02.645483 23805 log.cc:927] Time spent T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20251023 02:05:02.633386 23737 raft_consensus.cc:697] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:02.648325 23778 raft_consensus.cc:3055] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:02.648708 23778 raft_consensus.cc:740] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:02.650417 23784 log.cc:927] Time spent T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.715s user 0.002s sys 0.001s
I20251023 02:05:02.649895 23779 consensus_queue.cc:1059] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:05:02.649564 23778 consensus_queue.cc:260] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.652084 23754 raft_consensus.cc:2749] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:02.652590 23778 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:02.653944 23771 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.653657 23762 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.654668 23799 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.655772 23762 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:02.656589 23780 raft_consensus.cc:3055] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:02.656900 23780 raft_consensus.cc:740] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:02.657500 23780 consensus_queue.cc:260] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.658354 23780 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:02.662552 23791 raft_consensus.cc:3055] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:02.551290 21928 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:02.695432 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1009) took 2029 ms. Trace:
I20251023 02:05:02.555167 21957 raft_consensus.cc:1240] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20251023 02:05:02.696900 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 145643us
I20251023 02:05:02.534616 21953 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:02.698132 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 113) took 2030 ms. Trace:
I20251023 02:05:02.702389 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1285) took 2035 ms. Trace:
I20251023 02:05:02.702864 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 114) took 2035 ms. Trace:
W20251023 02:05:02.557135 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 46026.5us
W20251023 02:05:02.565431 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 49839.4us
I20251023 02:05:02.710465 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:02.710965 22102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1246) took 2047 ms (client timeout 2585 ms). Trace:
I20251023 02:05:02.589229 22110 rpcz_store.cc:276] 1023 02:05:00.664741 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.577078 (+1912337us) service_pool.cc:225] Handling call
1023 02:05:02.577331 (+ 253us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.578780 (+ 1449us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.578924 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.729780 22000 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:02.604857 22111 rpcz_store.cc:276] 1023 02:05:01.032328 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.602129 (+1569801us) service_pool.cc:225] Handling call
1023 02:05:02.602350 (+ 221us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.603574 (+ 1224us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.603750 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.713546 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1331) took 1833 ms. Trace:
I20251023 02:05:02.746610 22106 rpcz_store.cc:276] 1023 02:05:00.880068 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.592487 (+1712419us) service_pool.cc:225] Handling call
1023 02:05:02.592799 (+ 312us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.712835 (+120036us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.713234 (+ 399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.747926 22110 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:02.747836 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:02.748420 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1337) took 1679 ms. Trace:
I20251023 02:05:02.748535 22106 raft_consensus.cc:1240] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:02.748765 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1336) took 1689 ms. Trace:
I20251023 02:05:02.726971 21957 rpcz_store.cc:276] 1023 02:05:00.667356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.534514 (+1867158us) service_pool.cc:225] Handling call
1023 02:05:02.554989 (+ 20475us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:02.697415 (+142426us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.697926 (+ 511us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.710976 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.727869 21961 rpcz_store.cc:276] 1023 02:05:00.667632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.700694 (+2033062us) service_pool.cc:225] Handling call
1023 02:05:02.701558 (+ 864us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.702209 (+ 651us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.702266 (+ 57us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.702341 (+ 75us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:02.702392 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.702467 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.702614 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.748912 22111 rpcz_store.cc:276] 1023 02:05:01.069005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.746660 (+1677655us) service_pool.cc:225] Handling call
1023 02:05:02.747450 (+ 790us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.747946 (+ 496us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.747972 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.748026 (+ 54us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:02.748059 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.748120 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.748217 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42880}
I20251023 02:05:02.721829 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.721045 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.696303 21956 rpcz_store.cc:276] 1023 02:05:00.665182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.810417 (+145235us) service_pool.cc:225] Handling call
1023 02:05:01.818391 (+1007974us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:01.818977 (+ 586us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.694597 (+875620us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b54000bf4d0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:02.695069 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2272993536}
I20251023 02:05:02.663018 23791 raft_consensus.cc:740] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:02.745514 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:02.719630 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 133) took 1031 ms. Trace:
I20251023 02:05:02.721134 23297 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:02.723773 23569 raft_consensus.cc:3055] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:02.745447 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.714699 22103 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20251023 02:05:02.727152 21958 rpcz_store.cc:276] 1023 02:05:00.667108 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.460556 (+1793448us) service_pool.cc:225] Handling call
1023 02:05:02.460986 (+ 430us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:02.461536 (+ 550us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.461567 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:02.464876 (+ 3309us) log.cc:844] Serialized 26 byte log entry
1023 02:05:02.506263 (+ 41387us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:02.506322 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.506413 (+ 91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:02.701818 (+195405us) raft_consensus.cc:1685] finished
1023 02:05:02.701870 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.702204 (+ 334us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2880,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":5772,"thread_start_us":19871,"threads_started":1,"wal-append.queue_time_us":87322}
I20251023 02:05:02.752096 22110 rpcz_store.cc:276] 1023 02:05:01.059366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.746598 (+1687232us) service_pool.cc:225] Handling call
1023 02:05:02.747385 (+ 787us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.748488 (+ 1103us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.748684 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":100864}
I20251023 02:05:02.766996 22029 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:02.749274 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1339) took 1502 ms. Trace:
I20251023 02:05:02.769853 22106 rpcz_store.cc:276] 1023 02:05:01.247097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.747756 (+1500659us) service_pool.cc:225] Handling call
1023 02:05:02.748237 (+ 481us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.749058 (+ 821us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.749228 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.774363 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:02.775260 23771 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:02.768535 23569 raft_consensus.cc:740] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:02.776104 21961 raft_consensus.cc:1240] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20251023 02:05:02.604790 23690 consensus_queue.cc:237] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.761606 22036 rpcz_store.cc:276] 1023 02:05:01.687795 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:01.688980 (+ 1185us) service_pool.cc:225] Handling call
1023 02:05:01.689336 (+ 356us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:01.950718 (+261382us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.718922 (+768204us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400280854. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:02.719339 (+ 417us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":593473792}
I20251023 02:05:02.777401 22111 raft_consensus.cc:1240] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:05:02.782076 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.783350 22034 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:02.784866 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:02.788564 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:02.789428 22107 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1321) took 2308 ms (client timeout 1945 ms). Trace:
I20251023 02:05:02.789748 22029 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
W20251023 02:05:02.789885 22107 rpcz_store.cc:269] 1023 02:05:00.480738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.548995 (+ 68257us) service_pool.cc:225] Handling call
1023 02:05:02.788930 (+2239935us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b540008f7d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:02.789171 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2276104064}
I20251023 02:05:02.790577 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.767454 23791 consensus_queue.cc:260] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.790990 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:02.854146 23780 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:02.854769 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.780079 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1288) took 2111 ms. Trace:
W20251023 02:05:02.741936 22102 rpcz_store.cc:269] 1023 02:05:00.663132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:01.432318 (+769186us) service_pool.cc:225] Handling call
1023 02:05:02.710307 (+1277989us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54000fbc54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:02.710597 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1612500864}
I20251023 02:05:02.860934 21961 rpcz_store.cc:276] 1023 02:05:00.667892 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.770235 (+2102343us) service_pool.cc:225] Handling call
1023 02:05:02.772247 (+ 2012us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:02.779198 (+ 6951us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.779485 (+ 287us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:02.850173 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:02.854883 23569 consensus_queue.cc:260] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:02.863070 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 71554.2us
I20251023 02:05:02.791383 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1340) took 1539 ms. Trace:
I20251023 02:05:02.870471 22111 rpcz_store.cc:276] 1023 02:05:01.247422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.761283 (+1513861us) service_pool.cc:225] Handling call
1023 02:05:02.776991 (+ 15708us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.786697 (+ 9706us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.786883 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:02.873836 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 144549us
I20251023 02:05:02.878921 23593 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:05:02.877904 23818 consensus_queue.cc:1059] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
W20251023 02:05:02.881736 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:02.882341 23818 raft_consensus.cc:3055] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:02.883361 23818 raft_consensus.cc:740] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:02.882645 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:02.891937 22034 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:02.882437 23569 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:02.854766 23780 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:02.905979 23780 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:02.906929 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1295) took 1779 ms. Trace:
I20251023 02:05:02.907315 21958 rpcz_store.cc:276] 1023 02:05:01.127027 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.832393 (+1705366us) service_pool.cc:225] Handling call
1023 02:05:02.906210 (+ 73817us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.906576 (+ 366us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.906616 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.906672 (+ 56us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:02.906702 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.906751 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.906855 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.897899 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.910252 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:02.910686 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 116) took 2066 ms. Trace:
I20251023 02:05:02.911609 22033 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:02.912588 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1293) took 1786 ms. Trace:
W20251023 02:05:02.913530 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:02.913686 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:02.914798 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:02.882879 23691 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:02.902542 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 146) took 1353 ms. Trace:
I20251023 02:05:02.911130 21956 rpcz_store.cc:276] 1023 02:05:00.844779 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.787119 (+1942340us) service_pool.cc:225] Handling call
1023 02:05:02.893474 (+106355us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.910351 (+ 16877us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.910375 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.910415 (+ 40us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:02.910473 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.910526 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.910635 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.921963 23691 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:02.923030 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1297) took 1795 ms. Trace:
I20251023 02:05:02.923465 21956 rpcz_store.cc:276] 1023 02:05:01.127658 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.921731 (+1794073us) service_pool.cc:225] Handling call
1023 02:05:02.922294 (+ 563us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.922608 (+ 314us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.922639 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.922712 (+ 73us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:02.922778 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.922829 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.922962 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.930567 22036 raft_consensus.cc:1240] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:02.919860 22106 rpcz_store.cc:276] 1023 02:05:01.549600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.795609 (+1246009us) service_pool.cc:225] Handling call
1023 02:05:02.853823 (+ 58214us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.854487 (+ 664us) raft_consensus.cc:1534] Early marking committed up to index 6
1023 02:05:02.854943 (+ 456us) log.cc:844] Serialized 14 byte log entry
1023 02:05:02.902187 (+ 47244us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.902232 (+ 45us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:02.902268 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.902350 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.902488 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":10302,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":7115,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":241,"thread_start_us":38929,"threads_started":1}
I20251023 02:05:02.924552 23691 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:02.935536 22621 raft_consensus.cc:3055] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:02.935994 22621 raft_consensus.cc:740] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:02.936565 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:02.937047 22621 consensus_queue.cc:260] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.937824 22615 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251023 02:05:02.938441 22621 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:02.940783 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:02.777331 23737 consensus_queue.cc:237] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.913031 21953 rpcz_store.cc:276] 1023 02:05:01.126428 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.777760 (+1651332us) service_pool.cc:225] Handling call
1023 02:05:02.870383 (+ 92623us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.912025 (+ 41642us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.912068 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.912132 (+ 64us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:02.912174 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.912249 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.912369 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.928006 23791 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:02.886090 23818 consensus_queue.cc:260] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:02.954389 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:02.929713 22107 raft_consensus.cc:2955] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:02.956729 23774 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:02.960832 23774 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:02.963399 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:02.931098 22032 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
W20251023 02:05:02.966590 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:02.948623 23703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.647s user 0.034s sys 0.045s
I20251023 02:05:02.965428 23771 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:02.968118 23669 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:02.978405 22037 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:05:02.979169 23669 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:02.979821 22029 raft_consensus.cc:1275] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:02.953824 23779 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
W20251023 02:05:02.968590 23703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.837s user 0.035s sys 0.045s
I20251023 02:05:02.984974 22034 raft_consensus.cc:1275] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251023 02:05:02.983843 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 149) took 1433 ms. Trace:
I20251023 02:05:02.989087 22621 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251023 02:05:02.989614 22111 rpcz_store.cc:276] 1023 02:05:01.550506 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.900829 (+1350323us) service_pool.cc:225] Handling call
1023 02:05:02.959959 (+ 59130us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.983432 (+ 23473us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.983454 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.983502 (+ 48us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:02.983532 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.983585 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.983793 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:02.996945 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1298) took 1860 ms. Trace:
W20251023 02:05:02.999187 23741 log.cc:927] Time spent T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 1.857s user 0.000s sys 0.018s
I20251023 02:05:02.999491 21956 rpcz_store.cc:276] 1023 02:05:01.127951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.928828 (+1800877us) service_pool.cc:225] Handling call
1023 02:05:02.963175 (+ 34347us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.988339 (+ 25164us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.988358 (+ 19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.988386 (+ 28us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:02.988404 (+ 18us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.988441 (+ 37us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.988510 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.001520 21956 pending_rounds.cc:85] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 1
I20251023 02:05:03.001849 21956 pending_rounds.cc:107] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251023 02:05:03.002127 21956 raft_consensus.cc:2889] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:02.973734 23771 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.009281 22617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 249) took 1794 ms. Trace:
I20251023 02:05:03.009760 22617 rpcz_store.cc:276] 1023 02:05:01.214334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:01.216494 (+ 2160us) service_pool.cc:225] Handling call
1023 02:05:01.225702 (+ 9208us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.593819 (+1368117us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.008811 (+414992us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54001531d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:03.009104 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2613266304}
I20251023 02:05:03.016450 21961 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:03.018457 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1299) took 1890 ms. Trace:
I20251023 02:05:03.018734 21961 rpcz_store.cc:276] 1023 02:05:01.128233 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.939638 (+1811405us) service_pool.cc:225] Handling call
1023 02:05:02.972309 (+ 32671us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.016904 (+ 44595us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.018415 (+ 1511us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.017762 21928 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:03.007822 22002 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.032995 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1281) took 2292 ms. Trace:
W20251023 02:05:03.041796 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 89927.8us
I20251023 02:05:03.051496 23739 consensus_queue.cc:1059] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20251023 02:05:03.052712 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1287) took 2347 ms. Trace:
I20251023 02:05:03.053078 23005 raft_consensus.cc:3055] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:05:03.053818 23005 raft_consensus.cc:740] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:03.053186 21962 rpcz_store.cc:276] 1023 02:05:00.667632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.704642 (+2037010us) service_pool.cc:225] Handling call
1023 02:05:02.705223 (+ 581us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:02.705946 (+ 723us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:02.705981 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:02.706956 (+ 975us) log.cc:844] Serialized 52 byte log entry
1023 02:05:02.735055 (+ 28099us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:05:02.735099 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.737744 (+ 2645us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:02.936931 (+199187us) spinlock_profiling.cc:243] Waited 8.42 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:03.014072 (+ 77141us) raft_consensus.cc:1685] finished
1023 02:05:03.014104 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.014940 (+ 836us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":26516,"raft.run_cpu_time_us":455,"raft.run_wall_time_us":1826,"spinlock_wait_cycles":19367168,"thread_start_us":27592,"threads_started":1,"wal-append.queue_time_us":104757}
I20251023 02:05:03.055472 23005 consensus_queue.cc:260] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:03.059541 23753 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:05:03.060457 23646 consensus_meta.cc:287] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.590s user 0.006s sys 0.011s
I20251023 02:05:03.061367 22622 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:03.063432 23757 raft_consensus.cc:3055] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:03.063994 23757 raft_consensus.cc:740] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:03.071103 23669 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:03.071565 23828 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:03.072225 23828 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:03.072724 23828 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:03.073221 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 134) took 1385 ms. Trace:
I20251023 02:05:03.073666 22031 rpcz_store.cc:276] 1023 02:05:01.688172 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:01.689633 (+ 1461us) service_pool.cc:225] Handling call
1023 02:05:01.690016 (+ 383us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:01.831484 (+141468us) raft_consensus.cc:1534] Early marking committed up to index 6
1023 02:05:01.943908 (+112424us) log.cc:844] Serialized 14 byte log entry
1023 02:05:02.112609 (+168701us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:02.112667 (+ 58us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:02.112712 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.072808 (+960096us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b540020c1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:03.072871 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.073036 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24607,"prepare.queue_time_us":171323,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":26683,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"spinlock_wait_cycles":304788224,"thread_start_us":233840,"threads_started":2,"wal-append.queue_time_us":133762}
I20251023 02:05:03.076596 23407 raft_consensus.cc:3055] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:03.077121 23407 raft_consensus.cc:740] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:03.081522 22028 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:03.080210 21955 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
W20251023 02:05:03.084131 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.085772 22037 raft_consensus.cc:1240] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 5. Current term is 6. Ops: []
W20251023 02:05:03.087242 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 122704us
I20251023 02:05:03.088142 22002 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.090323 21928 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:03.092064 23703 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:03.092507 23703 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:03.092936 23703 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:05:03.093871 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 56283.1us
W20251023 02:05:03.094097 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.094945 22000 leader_election.cc:400] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4.
W20251023 02:05:03.097692 23541 consensus_meta.cc:287] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.509s user 0.005s sys 0.011s
I20251023 02:05:03.065382 23757 consensus_queue.cc:260] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.098320 23729 raft_consensus.cc:3055] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:05:03.098541 23757 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:03.098865 23729 raft_consensus.cc:740] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:03.100317 23779 raft_consensus.cc:2749] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:03.100066 23729 consensus_queue.cc:260] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.098235 23541 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.035144 21960 rpcz_store.cc:276] 1023 02:05:00.666127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:00.817504 (+151377us) service_pool.cc:225] Handling call
1023 02:05:00.817849 (+ 345us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:00.818173 (+ 324us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:00.818202 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:00.818628 (+ 426us) log.cc:844] Serialized 26 byte log entry
1023 02:05:01.255989 (+437361us) spinlock_profiling.cc:243] Waited 7.94 ms on lock 0x7b48000b22d8. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c649a7971 00007f9c6499ea37 00007f9c6499e8ca 00007f9c64a95a0f 00007f9c64a956d7 00007f9c64aa9cfc 00007f9c64aa9ca8 00007f9c64aa9c28 00007f9c64aa9be3 00007f9c64aa8e8c 00007f9c639e5326
1023 02:05:02.564586 (+1308597us) spinlock_profiling.cc:243] Waited 7.81 ms on lock 0x7b18003403c0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9cdd37 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c5f62f2bb 00007f9c5f634dc3 00007f9c5f635171 00007f9c5fa5465f
1023 02:05:02.897271 (+332685us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:02.926824 (+ 29553us) log.cc:844] Serialized 14 byte log entry
1023 02:05:02.941458 (+ 14634us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:02.950829 (+ 9371us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:02.955157 (+ 4328us) raft_consensus.cc:1685] finished
1023 02:05:02.955171 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:02.958902 (+ 3731us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8519,"prepare.queue_time_us":11466,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":73,"raft.queue_time_us":19188,"raft.run_cpu_time_us":3393,"raft.run_wall_time_us":435445,"spinlock_wait_cycles":36214400,"thread_start_us":240578,"threads_started":1,"wal-append.queue_time_us":2000389}
I20251023 02:05:03.022357 23780 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:03.112470 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.113626 22033 raft_consensus.cc:1240] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:03.078624 23407 consensus_queue.cc:260] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.111066 23729 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:03.131887 21961 raft_consensus.cc:1275] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251023 02:05:03.120102 23541 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:03.132400 21961 pending_rounds.cc:85] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 2
I20251023 02:05:03.137235 21961 pending_rounds.cc:107] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251023 02:05:03.138015 21961 raft_consensus.cc:2889] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:03.120127 23704 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20251023 02:05:03.141736 22111 raft_consensus.cc:1275] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:03.148151 23005 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:03.142940 21962 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:03.148214 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.152534 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1303) took 2023 ms. Trace:
I20251023 02:05:03.147574 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1345) took 1251 ms. Trace:
I20251023 02:05:03.153007 21962 rpcz_store.cc:276] 1023 02:05:01.129358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.072359 (+1943001us) service_pool.cc:225] Handling call
1023 02:05:03.128590 (+ 56231us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.152191 (+ 23601us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.152470 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.155804 23709 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:03.156651 23709 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:03.158491 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:03.159031 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1309) took 1444 ms. Trace:
I20251023 02:05:03.159554 21955 rpcz_store.cc:276] 1023 02:05:01.714462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.392269 (+677807us) service_pool.cc:225] Handling call
1023 02:05:03.158793 (+766524us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b54000c3d54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:03.158954 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":701359872}
I20251023 02:05:03.160135 21957 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:03.161476 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:03.163707 23736 consensus_queue.cc:1059] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
W20251023 02:05:03.155803 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.153801 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1302) took 2024 ms. Trace:
I20251023 02:05:03.166283 21961 rpcz_store.cc:276] 1023 02:05:01.129091 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.032514 (+1903423us) service_pool.cc:225] Handling call
1023 02:05:03.106235 (+ 73721us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.153561 (+ 47326us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.153751 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.167155 21955 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:05:03.161485 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 118) took 1849 ms. Trace:
I20251023 02:05:03.170099 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 120) took 1847 ms. Trace:
I20251023 02:05:03.170658 21955 rpcz_store.cc:276] 1023 02:05:01.320766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.160798 (+1840032us) service_pool.cc:225] Handling call
1023 02:05:03.161217 (+ 419us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.167454 (+ 6237us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.167579 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.170558 21960 rpcz_store.cc:276] 1023 02:05:01.312050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.126593 (+1814543us) service_pool.cc:225] Handling call
1023 02:05:03.155350 (+ 28757us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.161135 (+ 5785us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:03.161159 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:03.161221 (+ 62us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:03.161246 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.161328 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.161437 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.153519 22111 rpcz_store.cc:276] 1023 02:05:01.892329 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.017352 (+1125023us) service_pool.cc:225] Handling call
1023 02:05:03.048165 (+ 30813us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.143296 (+ 95131us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.144023 (+ 727us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:03.172712 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 43864.8us
I20251023 02:05:03.154197 23779 raft_consensus.cc:2804] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:03.178992 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 148) took 1626 ms. Trace:
I20251023 02:05:03.179368 22107 rpcz_store.cc:276] 1023 02:05:01.550203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.791073 (+1240870us) service_pool.cc:225] Handling call
1023 02:05:02.849030 (+ 57957us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.898438 (+ 49408us) raft_consensus.cc:1534] Early marking committed up to index 5
1023 02:05:03.020689 (+122251us) log.cc:844] Serialized 14 byte log entry
1023 02:05:03.175663 (+154974us) log.cc:844] Serialized 14 byte log entry
1023 02:05:03.175832 (+ 169us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:03.175881 (+ 49us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:03.175919 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.175982 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.176078 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":20247,"raft.queue_time_us":2469,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":4527,"thread_start_us":103725,"threads_started":1}
I20251023 02:05:03.168107 23709 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:03.172353 23407 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:03.165184 23736 raft_consensus.cc:3055] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:05:03.185695 23736 raft_consensus.cc:740] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:03.194559 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 730043us
I20251023 02:05:03.194947 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:03.194962 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.193327 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1296) took 2066 ms. Trace:
I20251023 02:05:03.197937 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.194972 22036 raft_consensus.cc:1275] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:03.195477 21962 raft_consensus.cc:1240] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:03.199728 22030 raft_consensus.cc:2393] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:03.201074 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 119) took 1885 ms. Trace:
I20251023 02:05:03.201400 21962 rpcz_store.cc:276] 1023 02:05:01.315350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.154650 (+1839300us) service_pool.cc:225] Handling call
1023 02:05:03.184331 (+ 29681us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.200835 (+ 16504us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.201022 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.202708 21955 raft_consensus.cc:1275] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:03.191058 23780 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:03.189862 23669 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20251023 02:05:03.205847 23816 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20251023 02:05:03.207127 22109 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251023 02:05:03.199370 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:03.210068 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1306) took 1791 ms. Trace:
I20251023 02:05:03.210613 21955 rpcz_store.cc:276] 1023 02:05:01.411773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.174507 (+1762734us) service_pool.cc:225] Handling call
1023 02:05:03.189572 (+ 15065us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.202970 (+ 13398us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.203554 (+ 584us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.197851 21958 rpcz_store.cc:276] 1023 02:05:01.127348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.908330 (+1780982us) service_pool.cc:225] Handling call
1023 02:05:02.908853 (+ 523us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:02.932207 (+ 23354us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:05:02.963235 (+ 31028us) log.cc:844] Serialized 14 byte log entry
1023 02:05:03.035049 (+ 71814us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:03.051189 (+ 16140us) log.cc:844] Serialized 52 byte log entry
1023 02:05:03.097161 (+ 45972us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:03.097385 (+ 224us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.097452 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:03.193108 (+ 95656us) raft_consensus.cc:1685] finished
1023 02:05:03.193137 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.193273 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64417,"prepare.queue_time_us":14913,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":61,"raft.queue_time_us":37984,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":7436,"thread_start_us":15830,"threads_started":1,"wal-append.queue_time_us":44275}
I20251023 02:05:03.212841 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1301) took 2084 ms. Trace:
I20251023 02:05:03.213122 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:03.190047 23593 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20251023 02:05:03.212375 21925 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.218144 22034 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20251023 02:05:03.220587 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 150) took 1669 ms. Trace:
I20251023 02:05:03.220994 22102 rpcz_store.cc:276] 1023 02:05:01.550777 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.902878 (+1352101us) service_pool.cc:225] Handling call
1023 02:05:03.211606 (+308728us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.219644 (+ 8038us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:03.219678 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:03.219728 (+ 50us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:03.219760 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.219830 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.220408 (+ 578us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b540032f5d0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:03.220515 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":555039488}
W20251023 02:05:03.225987 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.227085 21925 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.228174 21958 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20251023 02:05:03.228914 21925 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.227113 22029 raft_consensus.cc:1240] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:03.228945 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1023) took 1096 ms. Trace:
I20251023 02:05:03.230245 22030 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:03.230525 21958 rpcz_store.cc:276] 1023 02:05:02.132290 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.223275 (+1090985us) service_pool.cc:225] Handling call
1023 02:05:03.227520 (+ 4245us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:03.228653 (+ 1133us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.228895 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:03.231711 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.213531 21956 rpcz_store.cc:276] 1023 02:05:01.128818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.000675 (+1871857us) service_pool.cc:225] Handling call
1023 02:05:03.001175 (+ 500us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:03.003076 (+ 1901us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:03.003110 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:03.003698 (+ 588us) log.cc:844] Serialized 26 byte log entry
1023 02:05:03.110833 (+107135us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:03.135836 (+ 25003us) log.cc:844] Serialized 14 byte log entry
1023 02:05:03.139098 (+ 3262us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.139181 (+ 83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:03.212646 (+ 73465us) raft_consensus.cc:1685] finished
1023 02:05:03.212667 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.212787 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":24826,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":3646,"raft.queue_time_us":3760,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":7425,"thread_start_us":106284,"threads_started":1,"wal-append.queue_time_us":137436}
I20251023 02:05:03.235291 22111 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:03.213704 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.239820 21926 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
W20251023 02:05:03.241000 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.209533 23726 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:03.241987 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:03.247737 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.247884 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1308) took 1835 ms. Trace:
W20251023 02:05:03.248529 23725 consensus_meta.cc:287] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.601s user 0.011s sys 0.001s
I20251023 02:05:03.239552 23774 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:03.264467 23725 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:03.265833 22105 raft_consensus.cc:1240] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:03.266856 23725 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:03.268190 23799 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:03.270639 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1286) took 2603 ms. Trace:
I20251023 02:05:03.271114 21959 rpcz_store.cc:276] 1023 02:05:00.667356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.487755 (+1820399us) service_pool.cc:225] Handling call
1023 02:05:02.488141 (+ 386us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:02.703328 (+215187us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:05:02.742745 (+ 39417us) log.cc:844] Serialized 14 byte log entry
1023 02:05:03.120681 (+377936us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:03.120769 (+ 88us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:03.120804 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.270143 (+149339us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b540017d754. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:03.270203 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.270413 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":132080,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":7697,"raft.queue_time_us":6275,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":1235,"spinlock_wait_cycles":493630720,"thread_start_us":344588,"threads_started":2,"wal-append.queue_time_us":380152}
I20251023 02:05:03.251649 23297 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:03.277695 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:03.279795 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:03.282469 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 151) took 1680 ms. Trace:
I20251023 02:05:03.254928 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1305) took 1843 ms. Trace:
I20251023 02:05:03.254758 21960 rpcz_store.cc:276] 1023 02:05:01.412323 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.177345 (+1765022us) service_pool.cc:225] Handling call
1023 02:05:03.202369 (+ 25024us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.247556 (+ 45187us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:03.247579 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:03.247618 (+ 39us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:05:03.247664 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.247729 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.247834 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.259222 23713 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:03.263643 22034 raft_consensus.cc:1240] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251023 02:05:03.263682 22030 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:03.252753 22107 raft_consensus.cc:1240] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:03.284616 21961 rpcz_store.cc:276] 1023 02:05:01.411405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.169943 (+1758538us) service_pool.cc:225] Handling call
1023 02:05:03.200434 (+ 30491us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.248249 (+ 47815us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:03.248277 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:03.254665 (+ 6388us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:03.254706 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.254766 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.254878 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.283483 22106 rpcz_store.cc:276] 1023 02:05:01.585212 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.941926 (+1356714us) service_pool.cc:225] Handling call
1023 02:05:03.094774 (+152848us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9ae20e 000000000035a3c9 00007f9c649cd030 00007f9c64a95864 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:05:03.094966 (+ 192us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:03.095489 (+ 523us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:03.095525 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:03.096129 (+ 604us) log.cc:844] Serialized 26 byte log entry
1023 02:05:03.133643 (+ 37514us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:03.133681 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.133742 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:03.264766 (+131024us) raft_consensus.cc:1685] finished
1023 02:05:03.264795 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.265067 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":506,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":128,"spinlock_wait_cycles":7741184,"thread_start_us":37051,"threads_started":1,"wal-append.queue_time_us":100332}
I20251023 02:05:03.285961 23709 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:03.291210 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:03.293476 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.237056 23771 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:03.296211 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:03.297940 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:03.298206 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 64514.9us
W20251023 02:05:03.290035 23831 log.cc:927] Time spent T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
W20251023 02:05:03.303140 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 74389.8us
I20251023 02:05:03.244846 23593 consensus_queue.cc:1059] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20251023 02:05:03.305343 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:03.303355 23774 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251023 02:05:03.312721 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 60204.1us
I20251023 02:05:03.320909 23720 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:03.311980 23762 consensus_queue.cc:1059] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20251023 02:05:03.321424 23720 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:03.321573 23836 log.cc:927] Time spent T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20251023 02:05:03.321903 23720 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:03.322202 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:03.322363 23736 consensus_queue.cc:260] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.323763 23736 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:03.322842 22622 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:03.321911 23762 raft_consensus.cc:3055] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
W20251023 02:05:03.318706 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.314086 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39634 (request call id 851) took 1274 ms. Trace:
I20251023 02:05:03.327428 22109 rpcz_store.cc:276] 1023 02:05:02.039587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.208113 (+1168526us) service_pool.cc:225] Handling call
1023 02:05:03.314006 (+105893us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.331972 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1342) took 1440 ms. Trace:
I20251023 02:05:03.334969 21925 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.326334 23762 raft_consensus.cc:740] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:03.310819 23774 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:03.334985 22108 rpcz_store.cc:276] 1023 02:05:01.891471 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.025233 (+133762us) service_pool.cc:225] Handling call
1023 02:05:03.279370 (+1254137us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:05:03.331910 (+ 52540us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8720640}
I20251023 02:05:03.308002 22028 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251023 02:05:03.343997 23593 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:03.344673 22077 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.345510 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:03.346407 23753 consensus_queue.cc:1059] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251023 02:05:03.352419 22076 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.308544 22029 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:03.357306 23755 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:03.357658 22033 raft_consensus.cc:1275] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:05:03.360747 23753 raft_consensus.cc:3055] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:03.361153 23753 raft_consensus.cc:740] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:03.361832 21928 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:03.362071 23753 consensus_queue.cc:260] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:03.362720 23520 raft_consensus.cc:3055] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:03.363099 23520 raft_consensus.cc:740] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:03.363329 23753 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:03.365761 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 122) took 1059 ms. Trace:
I20251023 02:05:03.366037 21958 rpcz_store.cc:276] 1023 02:05:02.306390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.283022 (+976632us) service_pool.cc:225] Handling call
1023 02:05:03.314649 (+ 31627us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.353290 (+ 38641us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:03.353325 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:03.365546 (+ 12221us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:03.365573 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.365651 (+ 78us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.365722 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:03.367655 23593 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:03.368332 23833 log.cc:927] Time spent T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20251023 02:05:03.369735 23593 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:03.367668 21960 raft_consensus.cc:1240] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:03.371619 22002 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:03.373203 23663 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:03.373790 23663 raft_consensus.cc:697] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:03.374025 23726 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:05:03.374059 23780 consensus_queue.cc:1048] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20251023 02:05:03.309762 23709 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:03.381914 22107 raft_consensus.cc:1240] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:05:03.383255 23709 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:03.383778 23642 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:03.353041 23774 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:03.384320 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:03.384333 23642 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:03.390116 22109 raft_consensus.cc:1240] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251023 02:05:03.391469 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:03.392163 22621 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:03.396245 21956 raft_consensus.cc:1240] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:05:03.399524 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 147) took 1849 ms. Trace:
W20251023 02:05:03.402873 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.404160 22001 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:03.411867 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 57132.3us
I20251023 02:05:03.411957 23569 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:05:03.412593 22108 raft_consensus.cc:1275] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:05:03.413030 21928 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.400198 23737 consensus_queue.cc:1048] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.014s
I20251023 02:05:03.414407 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1316) took 2966 ms. Trace:
I20251023 02:05:03.414757 22103 rpcz_store.cc:276] 1023 02:05:00.447925 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.368887 (+1920962us) service_pool.cc:225] Handling call
1023 02:05:02.713502 (+344615us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:02.774171 (+ 60669us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.414194 (+640023us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b54000fbc50. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:03.414339 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":791585664}
I20251023 02:05:03.416332 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:03.417232 23473 raft_consensus.cc:3055] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:03.417642 23473 raft_consensus.cc:740] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:03.418399 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.414870 23771 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:03.419128 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:03.420024 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.399926 22110 rpcz_store.cc:276] 1023 02:05:01.549916 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.238240 (+1688324us) service_pool.cc:225] Handling call
1023 02:05:03.238669 (+ 429us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:03.289226 (+ 50557us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:03.289254 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:03.320986 (+ 31732us) log.cc:844] Serialized 26 byte log entry
1023 02:05:03.397431 (+ 76445us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:03.397471 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.397856 (+ 385us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:03.399285 (+ 1429us) raft_consensus.cc:1685] finished
1023 02:05:03.399301 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.399452 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17352,"raft.queue_time_us":50208,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":6555,"thread_start_us":75992,"threads_started":1,"wal-append.queue_time_us":76446}
I20251023 02:05:03.420554 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:03.424419 23799 raft_consensus.cc:2749] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:03.424321 23473 consensus_queue.cc:260] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.425392 22621 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251023 02:05:03.425350 23737 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:03.426151 23297 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:05:03.426229 22077 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:03.430749 23844 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election won for term 5
W20251023 02:05:03.434738 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 99913.3us
I20251023 02:05:03.364217 23520 consensus_queue.cc:260] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.407850 23757 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 02:05:03.401122 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.438385 21961 raft_consensus.cc:1275] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20251023 02:05:03.439610 23780 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251023 02:05:03.395779 23828 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.440466 23842 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:03.374877 23663 consensus_queue.cc:237] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.430663 23762 consensus_queue.cc:260] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:03.445003 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 113865us
W20251023 02:05:03.446817 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.456156 23844 raft_consensus.cc:697] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:03.446007 21961 pending_rounds.cc:85] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 1
I20251023 02:05:03.471729 21961 pending_rounds.cc:107] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251023 02:05:03.437964 23737 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:03.466012 21954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1022) took 1333 ms (client timeout 1535 ms). Trace:
I20251023 02:05:03.467324 23828 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:03.468266 23520 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:03.476734 21925 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.472092 21961 raft_consensus.cc:2889] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:03.406334 23729 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20251023 02:05:03.474673 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1313) took 1369 ms. Trace:
I20251023 02:05:03.469336 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1312) took 1665 ms. Trace:
I20251023 02:05:03.477561 21962 rpcz_store.cc:276] 1023 02:05:01.803529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.211998 (+1408469us) service_pool.cc:225] Handling call
1023 02:05:03.253237 (+ 41239us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:03.302985 (+ 49748us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:03.303019 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:03.336993 (+ 33974us) log.cc:844] Serialized 52 byte log entry
1023 02:05:03.445040 (+108047us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:03.468809 (+ 23769us) log.cc:844] Serialized 14 byte log entry
1023 02:05:03.469008 (+ 199us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.469074 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:03.469110 (+ 36us) raft_consensus.cc:1685] finished
1023 02:05:03.469127 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.469279 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24721,"prepare.queue_time_us":23324,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":47877,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":4686,"thread_start_us":54043,"threads_started":2,"wal-append.queue_time_us":99270}
I20251023 02:05:03.477473 21955 rpcz_store.cc:276] 1023 02:05:02.104985 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.225788 (+1120803us) service_pool.cc:225] Handling call
1023 02:05:03.302058 (+ 76270us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.334617 (+ 32559us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:05:03.384903 (+ 50286us) log.cc:844] Serialized 14 byte log entry
1023 02:05:03.474338 (+ 89435us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:03.474383 (+ 45us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:03.474446 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.474527 (+ 81us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:03.474627 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21808,"prepare.queue_time_us":40779,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":57628,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":75,"thread_start_us":63228,"threads_started":2,"wal-append.queue_time_us":25479}
W20251023 02:05:03.475160 21954 rpcz_store.cc:269] 1023 02:05:02.131999 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.353364 (+221365us) service_pool.cc:225] Handling call
1023 02:05:03.443056 (+1089692us) spinlock_profiling.cc:243] Waited 40.8 ms on lock 0x7b5400218754. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:03.465778 (+ 22722us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93754752}
I20251023 02:05:03.480772 23671 consensus_queue.cc:1059] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
W20251023 02:05:03.493034 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:03.496856 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.489989 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:03.476130 23737 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:03.497864 22000 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.501041 23721 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:03.502243 23844 consensus_queue.cc:237] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:03.503520 23840 log.cc:927] Time spent T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
W20251023 02:05:03.504313 23841 log.cc:927] Time spent T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20251023 02:05:03.504844 23845 consensus_queue.cc:1059] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20251023 02:05:03.505301 21925 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.505445 21953 raft_consensus.cc:1240] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:05:03.506785 23743 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:03.506517 23774 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:03.506503 23755 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:03.515468 22076 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.483059 23569 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:03.501111 23838 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
I20251023 02:05:03.508719 23671 consensus_queue.cc:1243] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251023 02:05:03.501067 23485 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:03.526746 23762 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:03.529680 23842 raft_consensus.cc:2749] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:05:03.530988 22076 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.519891 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1317) took 1086 ms. Trace:
I20251023 02:05:03.532095 21961 rpcz_store.cc:276] 1023 02:05:02.433423 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:03.365040 (+931617us) service_pool.cc:225] Handling call
1023 02:05:03.434536 (+ 69496us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.519648 (+ 85112us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.519838 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:03.533962 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 51852.3us
I20251023 02:05:03.534750 23521 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:03.537011 23799 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:03.537524 23799 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:03.537936 23799 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:03.537647 23726 consensus_queue.cc:1059] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:05:03.537158 22000 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.544831 22076 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.546399 22077 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:03.548055 23733 raft_consensus.cc:2804] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:05:03.548343 22077 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:03.548631 23733 raft_consensus.cc:697] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:03.549218 23727 consensus_queue.cc:1048] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:05:03.549760 23709 consensus_queue.cc:1048] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:05:03.550350 23732 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:03.550890 23732 raft_consensus.cc:697] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:03.551321 23774 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:03.551833 23732 consensus_queue.cc:237] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.553694 23727 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:03.555068 23791 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:03.555430 23727 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:03.522910 23721 raft_consensus.cc:697] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
W20251023 02:05:03.556502 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 51314.6us
I20251023 02:05:03.521638 23725 consensus_queue.cc:1059] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20251023 02:05:03.549793 23733 consensus_queue.cc:237] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.560283 23637 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:03.560876 23721 consensus_queue.cc:237] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.544503 23726 consensus_queue.cc:1243] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer ca1362b9d9b34073bf35b6f6c9d1159a log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20251023 02:05:03.566789 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.537096 23780 consensus_queue.cc:1048] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.021s
I20251023 02:05:03.525573 23569 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:03.571642 22110 raft_consensus.cc:1275] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251023 02:05:03.571475 23720 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.579381 23569 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:03.581388 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.582114 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:03.572821 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:03.568254 23709 consensus_queue.cc:1059] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.006s
I20251023 02:05:03.574126 22107 raft_consensus.cc:1240] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:03.589466 22033 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:03.594039 21928 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:03.580418 23720 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:03.598503 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:03.588871 23771 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:03.598124 21925 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.590950 23755 raft_consensus.cc:2749] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:03.612557 23854 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:03.613044 23673 consensus_queue.cc:1048] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20251023 02:05:03.619280 22076 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:03.619817 23848 log.cc:927] Time spent T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.123s user 0.000s sys 0.001s
I20251023 02:05:03.597862 23774 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:03.613092 23854 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.624009 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.624780 22033 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:03.625800 23755 consensus_queue.cc:1048] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:03.626734 23771 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251023 02:05:03.627732 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 59590.9us
I20251023 02:05:03.628546 21925 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.602823 23856 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:03.630563 21928 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:03.643163 22033 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:03.644917 22111 raft_consensus.cc:1275] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251023 02:05:03.650892 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:03.635938 23703 consensus_meta.cc:287] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.540s user 0.012s sys 0.000s
W20251023 02:05:03.662293 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.669996 23828 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:03.662840 23637 raft_consensus.cc:697] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:03.670476 23828 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:03.671036 23828 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:03.671082 23569 consensus_queue.cc:1059] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251023 02:05:03.670353 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:03.622382 23854 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:03.632790 23774 raft_consensus.cc:697] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:03.680258 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 104105us
I20251023 02:05:03.657857 23771 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251023 02:05:03.687214 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.671351 23637 consensus_queue.cc:237] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:03.690294 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 71146.5us
I20251023 02:05:03.704654 23725 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:03.705332 23725 raft_consensus.cc:697] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:03.686981 23771 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:03.714349 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.695343 23753 raft_consensus.cc:3055] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:03.674270 23703 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.685796 23854 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:03.727344 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1020) took 2635 ms. Trace:
I20251023 02:05:03.726385 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:03.727864 21957 rpcz_store.cc:276] 1023 02:05:01.091590 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.773519 (+1681929us) service_pool.cc:225] Handling call
1023 02:05:03.159820 (+386301us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:03.160676 (+ 856us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.724432 (+563756us) spinlock_profiling.cc:243] Waited 386 ms on lock 0x7b54000c3d50. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:03.727274 (+ 2842us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":886835968}
I20251023 02:05:03.693768 23774 consensus_queue.cc:237] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.733265 22077 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.730814 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:03.721849 23771 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:03.689666 23737 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:03.740087 22110 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:03.733808 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:03.685282 23856 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.713791 23727 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:03.721489 23753 raft_consensus.cc:740] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
W20251023 02:05:03.744982 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.749888 22104 raft_consensus.cc:2410] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251023 02:05:03.762705 23758 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:03.763202 23758 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.751037 21925 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.734860 23709 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:03.737282 23854 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:03.777206 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.778044 22111 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:05:03.766077 22037 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
W20251023 02:05:03.794466 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:03.783210 23855 log.cc:927] Time spent T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20251023 02:05:03.796324 23872 consensus_queue.cc:1059] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.067s
I20251023 02:05:03.801442 22077 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:03.783753 23844 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:03.802771 23864 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:03.803367 23865 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:03.803925 23865 raft_consensus.cc:697] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:03.804067 23854 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:03.804944 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.803380 23864 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:03.813648 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.814723 22110 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
W20251023 02:05:03.810216 22076 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.535s (SENT) [suppressed 38 similar messages]
W20251023 02:05:03.816807 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.766623 23713 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:03.820469 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:03.820710 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.819239 22032 raft_consensus.cc:2393] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
W20251023 02:05:03.826740 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 75853.7us
W20251023 02:05:03.827244 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.812759 23864 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:03.829061 21908 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet 6dddaeefc3d449a0ba9b3e6695e37507
I20251023 02:05:03.834031 22028 raft_consensus.cc:1240] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
W20251023 02:05:03.835358 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.784215 23671 consensus_queue.cc:1048] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20251023 02:05:03.780454 23709 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.767140 23725 consensus_queue.cc:237] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.841342 22075 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.841472 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:03.835953 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet 6dddaeefc3d449a0ba9b3e6695e37507
I20251023 02:05:03.842006 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.842774 23703 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:03.837136 23818 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:03.843575 23709 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:03.845605 23856 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:03.860327 23861 log.cc:927] Time spent T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
W20251023 02:05:03.860710 22104 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1327) took 3019 ms (client timeout 3363 ms). Trace:
W20251023 02:05:03.861169 22104 rpcz_store.cc:269] 1023 02:05:00.840959 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:01.972569 (+1131610us) service_pool.cc:225] Handling call
1023 02:05:03.860485 (+1887916us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b540022ef54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:03.860651 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3843706624}
I20251023 02:05:03.861815 23737 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:03.863157 23828 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.865248 23828 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:03.866196 21908 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet 5e923b66ccf04dd8985bbfe31adbf0b4
W20251023 02:05:03.866488 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet 5e923b66ccf04dd8985bbfe31adbf0b4
I20251023 02:05:03.868023 23642 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:03.869609 23864 consensus_queue.cc:1059] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.013s
I20251023 02:05:03.822535 23753 consensus_queue.cc:260] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:03.871402 23753 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:03.875547 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 49512.4us
I20251023 02:05:03.875034 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.880314 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.880296 22036 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:03.885174 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:03.885850 22621 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:03.888062 21928 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:03.887091 21925 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.893468 21925 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:03.894465 23856 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:03.894934 23856 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:03.893528 23757 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251023 02:05:03.909492 23867 log.cc:927] Time spent T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20251023 02:05:03.898880 22000 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:03.804971 23865 consensus_queue.cc:237] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.895323 23856 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:05:03.924342 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 114162us
I20251023 02:05:03.907532 23703 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:03.907332 23828 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:03.926265 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:03.917150 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:03.943274 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 70614.5us
I20251023 02:05:03.944072 22077 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:03.936374 23737 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:03.926090 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:03.948374 22589 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
I20251023 02:05:03.949621 23799 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:03.951882 23758 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:03.953933 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.954977 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:03.956501 23779 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:03.957010 22034 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:03.958568 23569 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:05:03.965633 23868 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:03.967229 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:03.995123 22036 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:03.970499 22621 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:03.937045 23828 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:03.990020 22034 raft_consensus.cc:606] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:03.997944 22034 raft_consensus.cc:614] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:03.990655 22109 raft_consensus.cc:2393] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
W20251023 02:05:03.996794 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:03.996250 22077 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:03.998842 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 72873.8us
I20251023 02:05:03.997763 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1334) took 2965 ms. Trace:
I20251023 02:05:04.000162 22105 rpcz_store.cc:276] 1023 02:05:01.027638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.599915 (+1572277us) service_pool.cc:225] Handling call
1023 02:05:02.600194 (+ 279us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.266258 (+666064us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:03.978364 (+712106us) spinlock_profiling.cc:243] Waited 665 ms on lock 0x7b540022f454. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:03.992759 (+ 14395us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1530077440}
I20251023 02:05:04.004427 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.005605 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:04.006322 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.006582 22034 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:04.007117 22034 raft_consensus.cc:606] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:04.007432 22034 raft_consensus.cc:614] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:04.007511 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.008255 22102 raft_consensus.cc:2393] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:04.008792 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251023 02:05:04.009091 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.015218 22001 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:04.001039 23828 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:03.993666 23779 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:05:04.023360 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.014161 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:05:04.032649 23799 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.032974 23687 consensus_queue.cc:1059] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20251023 02:05:04.033461 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:04.002547 23758 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:04.034600 23737 raft_consensus.cc:3055] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:04.040544 23406 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:04.040333 23771 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.044729 22037 raft_consensus.cc:1275] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:04.046594 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.063789 22109 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
W20251023 02:05:04.047483 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 91196us
I20251023 02:05:04.048406 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.030009 23703 raft_consensus.cc:2749] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:04.079236 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:04.081362 22109 raft_consensus.cc:2393] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:04.049047 23758 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:04.084604 22589 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:04.049299 23737 raft_consensus.cc:740] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:04.094784 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:04.097735 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.070230 23406 raft_consensus.cc:697] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:04.089809 23642 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:05:04.114565 23886 log.cc:927] Time spent T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20251023 02:05:04.115724 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:04.116930 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.117969 23743 raft_consensus.cc:3055] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:04.118146 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.118371 23743 raft_consensus.cc:740] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:04.119194 23743 consensus_queue.cc:260] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:04.119812 22105 raft_consensus.cc:1275] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251023 02:05:04.120150 22105 pending_rounds.cc:85] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 2
I20251023 02:05:04.120424 22105 pending_rounds.cc:107] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251023 02:05:04.120672 22105 raft_consensus.cc:2889] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:04.121841 22105 raft_consensus.cc:1240] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:04.088551 23758 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:04.125921 23737 consensus_queue.cc:260] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:04.126421 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.126729 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.092736 23733 consensus_queue.cc:1059] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20251023 02:05:04.127804 21961 raft_consensus.cc:2393] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:04.091934 23872 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.129686 21957 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251023 02:05:04.129874 23872 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.087615 23709 consensus_queue.cc:1048] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20251023 02:05:04.133381 23406 consensus_queue.cc:237] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:04.120432 23743 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:04.135161 23737 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:04.140090 22109 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:04.148712 23727 raft_consensus.cc:3055] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
W20251023 02:05:04.152925 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:04.158088 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.149315 23727 raft_consensus.cc:740] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:04.169407 23819 raft_consensus.cc:3055] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:04.169821 23727 consensus_queue.cc:260] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.170893 23727 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:04.162210 21955 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251023 02:05:04.162750 23774 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.171773 23774 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.172636 23385 raft_consensus.cc:3055] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:04.172993 23385 raft_consensus.cc:740] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:04.162070 23703 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.173786 23385 consensus_queue.cc:260] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.174028 23703 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.174841 23385 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:04.176021 23703 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.169833 23819 raft_consensus.cc:740] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:04.178525 23819 consensus_queue.cc:260] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.179481 23819 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:04.162819 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.181986 21961 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:04.182921 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1300) took 3054 ms. Trace:
I20251023 02:05:04.183023 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:04.183359 21953 rpcz_store.cc:276] 1023 02:05:01.128533 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:02.958653 (+1830120us) service_pool.cc:225] Handling call
1023 02:05:03.460715 (+502062us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:03.522027 (+ 61312us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:04.177345 (+655318us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b5400218750. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:04.177557 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1009753472}
I20251023 02:05:04.163901 23739 raft_consensus.cc:3055] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:04.184795 23739 raft_consensus.cc:740] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:04.164031 23845 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.185387 22032 raft_consensus.cc:1275] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:05:04.185806 23845 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.186206 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.186462 22105 raft_consensus.cc:3055] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:04.186823 22102 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:04.164314 23828 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:04.163457 23799 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.188480 22105 raft_consensus.cc:740] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:04.188315 23799 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.189312 23828 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:04.189635 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 78054.5us
I20251023 02:05:04.189951 23828 consensus_queue.cc:1048] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:04.190359 23799 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.166559 23713 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.166658 22111 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20251023 02:05:04.167110 21958 raft_consensus.cc:1240] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:04.165457 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:04.170727 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.167146 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.192906 21956 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:04.192544 21962 raft_consensus.cc:2368] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5.
I20251023 02:05:04.193328 21957 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:04.200811 23521 consensus_queue.cc:1048] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:05:04.204744 21954 raft_consensus.cc:1240] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:04.152058 23856 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.206519 21960 raft_consensus.cc:1240] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 4. Ops: []
I20251023 02:05:04.206555 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:04.192003 23739 consensus_queue.cc:260] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.208186 23757 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:04.208678 23757 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:04.208740 23739 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:04.189308 22105 consensus_queue.cc:260] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.213069 22030 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:04.213747 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.217039 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:04.225602 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:04.191293 23713 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.229398 22105 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:04.233424 22034 raft_consensus.cc:1240] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20251023 02:05:04.238210 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:04.238610 21958 raft_consensus.cc:1240] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:04.238835 21957 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:04.236164 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:04.240067 22032 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
W20251023 02:05:04.241712 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.246507 21928 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:04.113734 23888 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:04.158142 23891 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:04.258433 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.260259 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:04.284904 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 169452us
I20251023 02:05:04.285215 21958 raft_consensus.cc:1240] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20251023 02:05:04.238090 23845 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.293174 21959 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:04.293736 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:04.301324 22032 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:04.294759 21962 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:04.303468 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:04.295724 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:04.307085 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 222624us
W20251023 02:05:04.308164 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.268710 21956 raft_consensus.cc:2393] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:05:04.294375 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.341753 22032 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:04.340443 23872 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:04.269125 23703 raft_consensus.cc:2804] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:04.237440 23779 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.268544 23891 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:04.348212 21962 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
W20251023 02:05:04.339156 23874 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20251023 02:05:04.350800 21928 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:04.292802 23774 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.290783 23888 raft_consensus.cc:697] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:04.368930 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:04.370888 21958 raft_consensus.cc:2410] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251023 02:05:04.359159 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:04.366888 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:04.369279 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.354368 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.359076 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:04.341506 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.374895 21953 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:04.375437 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.376906 22037 raft_consensus.cc:1240] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:04.379125 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.379797 23726 raft_consensus.cc:1064] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:05:04.380251 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:04.380992 21959 raft_consensus.cc:2393] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:04.370026 23758 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.386703 23758 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:04.387848 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:04.379168 23407 raft_consensus.cc:1064] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:05:04.356374 23703 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:04.389205 21962 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:05:04.387689 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:04.386850 22036 raft_consensus.cc:1275] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251023 02:05:04.372825 22032 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:04.388763 21959 raft_consensus.cc:2376] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2. Re-sending same reply.
I20251023 02:05:04.413770 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.414281 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.415202 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.360395 23891 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:04.424615 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.372349 23845 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:04.429502 21962 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:04.430629 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:04.430835 21958 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:05:04.401649 23856 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:04.439435 23889 log.cc:927] Time spent T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20251023 02:05:04.439999 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:04.440804 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.443969 22036 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:04.405273 23703 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:04.448719 21953 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:04.411055 23771 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.454694 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.442839 23687 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.460346 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.467700 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:04.443310 23845 raft_consensus.cc:697] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:04.467170 22032 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:04.475925 21928 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:04.468569 22031 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:04.430198 23721 consensus_queue.cc:1048] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20251023 02:05:04.451716 23713 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:04.487293 22077 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:04.459800 21962 raft_consensus.cc:1240] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251023 02:05:04.460563 21960 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:04.489741 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:04.490350 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 91574.8us
I20251023 02:05:04.490571 23005 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.491024 23005 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.499577 21960 raft_consensus.cc:1275] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251023 02:05:04.500023 21960 pending_rounds.cc:85] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 1
I20251023 02:05:04.500250 21960 pending_rounds.cc:107] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251023 02:05:04.500511 21960 raft_consensus.cc:2889] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251023 02:05:04.501694 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.504422 23005 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.488456 23771 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election won for term 5
W20251023 02:05:04.507517 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 109109us
W20251023 02:05:04.508677 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 134368us
I20251023 02:05:04.032563 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:05:04.501516 22109 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
W20251023 02:05:04.518852 23899 log.cc:927] Time spent T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20251023 02:05:04.496714 22028 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
W20251023 02:05:04.521929 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.498575 23713 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:04.473173 23687 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.522634 21962 raft_consensus.cc:1240] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:04.527050 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.528383 22031 raft_consensus.cc:1275] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:05:04.530516 22036 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:04.536656 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.538736 23898 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.105s
I20251023 02:05:04.541679 23743 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.542411 23743 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.544597 23743 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:04.546838 23687 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.531435 23713 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:04.547747 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.547806 23713 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:04.526288 23771 raft_consensus.cc:697] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:04.554734 23897 log.cc:927] Time spent T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20251023 02:05:04.554524 23771 consensus_queue.cc:237] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.565346 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.518954 23845 consensus_queue.cc:237] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.524574 23865 consensus_queue.cc:1059] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20251023 02:05:04.581727 22103 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
W20251023 02:05:04.582283 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 48796.8us
I20251023 02:05:04.582755 21956 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
W20251023 02:05:04.584358 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 44083.6us
I20251023 02:05:04.586287 21957 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:04.644749 21959 raft_consensus.cc:1240] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:04.599926 22105 raft_consensus.cc:1275] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20251023 02:05:04.646072 22105 pending_rounds.cc:85] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 2
I20251023 02:05:04.646728 22105 pending_rounds.cc:107] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251023 02:05:04.593011 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:04.647341 22105 raft_consensus.cc:2889] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:04.586223 23297 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.483049 23888 consensus_queue.cc:237] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:04.583556 23856 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.583899 23779 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.493927 23726 consensus_queue.cc:237] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:04.647508 21960 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:04.652488 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 52612.8us
W20251023 02:05:04.652609 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.651244 22108 raft_consensus.cc:1240] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:04.654196 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.657071 22077 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:04.650656 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:04.703071 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 46103.6us
W20251023 02:05:04.657160 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 74689.8us
I20251023 02:05:04.663209 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:04.706269 22031 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:04.673728 22002 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:04.688915 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.691864 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.653561 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:04.711119 22621 raft_consensus.cc:2393] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:04.691304 21953 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:04.702298 22103 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:04.715270 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.717047 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:04.717751 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.662026 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:04.749787 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:04.750000 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 76412.1us
I20251023 02:05:04.750602 21953 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:05:04.751693 21956 raft_consensus.cc:2393] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:04.751533 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.754765 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:04.766568 21925 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:04.643074 23569 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:04.768500 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.770351 21955 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:04.773257 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:04.773839 22110 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:04.659260 23725 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:04.775146 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 58473us
W20251023 02:05:04.662364 23717 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20251023 02:05:04.654666 23785 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.679682 23842 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.775254 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.780066 22000 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:04.673302 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:04.781405 21957 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:04.667902 23733 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:04.719878 23869 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:04.647799 23799 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:04.757332 23816 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.670867 23762 consensus_queue.cc:1048] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
I20251023 02:05:04.684687 23779 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.097s
I20251023 02:05:04.791656 22077 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:04.792984 21956 raft_consensus.cc:1275] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:04.673110 23406 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.108s
I20251023 02:05:04.672431 23856 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.660534 23297 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.780544 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:04.712106 23703 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.663688 23757 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
W20251023 02:05:04.818564 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:04.820000 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.816399 21926 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:04.830328 23547 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:04.830790 23547 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:04.831122 23547 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:04.833205 23819 consensus_meta.cc:287] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.653s user 0.014s sys 0.000s
I20251023 02:05:04.791098 23785 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.835312 23785 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.781005 23569 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:04.837100 23906 log.cc:927] Time spent T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20251023 02:05:04.839007 23297 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.839291 23920 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:04.839524 23297 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.839892 23920 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.839967 23297 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:04.840387 23920 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:04.798681 23671 raft_consensus.cc:2804] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:04.841787 23671 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:04.842585 21953 raft_consensus.cc:1240] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:04.842109 23713 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.846694 22110 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:04.847322 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:04.842697 23703 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.791925 23733 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:04.849877 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 66537.1us
I20251023 02:05:04.849917 21956 raft_consensus.cc:1240] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:04.799204 23869 raft_consensus.cc:697] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:04.854389 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:04.855000 22032 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:04.859611 21954 raft_consensus.cc:1275] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:04.794939 23755 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:04.861073 22622 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251023 02:05:04.820868 23872 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:04.861285 23771 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:04.862509 21928 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:04.863492 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:04.866027 22002 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:04.783144 23725 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:04.862965 23914 log.cc:927] Time spent T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20251023 02:05:04.837261 23856 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.842141 23671 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:04.871080 23757 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:04.892309 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:04.820662 23799 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.874979 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:04.870813 22621 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEARNER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251023 02:05:04.872501 23872 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:04.796533 23816 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.796792 23731 consensus_queue.cc:1048] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.270s
I20251023 02:05:04.805241 23842 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:04.898406 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 131975us
I20251023 02:05:04.890205 23005 consensus_queue.cc:1059] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251023 02:05:04.875165 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.899696 23005 consensus_queue.cc:1243] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251023 02:05:04.891893 22002 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:04.869377 21960 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
W20251023 02:05:04.897790 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 241127us
I20251023 02:05:04.894840 21956 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:04.910800 23005 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:04.915313 22002 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:04.897435 23703 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251023 02:05:04.919071 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:04.875916 23819 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20251023 02:05:04.943217 21958 raft_consensus.cc:1275] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:05:04.873875 23755 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:04.901665 23297 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:04.914258 23757 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:04.943783 23799 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.926288 23923 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:04.926632 23727 consensus_meta.cc:287] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.744s user 0.005s sys 0.005s
I20251023 02:05:04.927209 21956 raft_consensus.cc:1275] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20251023 02:05:04.929358 23905 log.cc:927] Time spent T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.443s user 0.000s sys 0.004s
W20251023 02:05:04.959750 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 59930.9us
I20251023 02:05:04.933791 21959 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:04.934775 23816 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:04.936618 23743 raft_consensus.cc:3055] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:05:04.962996 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:04.945936 23703 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:04.966042 22033 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:04.966428 21925 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:04.900840 23736 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20251023 02:05:04.937970 23920 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:04.939242 23725 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:04.940202 22107 raft_consensus.cc:1275] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:05:04.941195 23713 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:04.943104 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:04.920459 23689 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:04.898650 23872 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:04.949236 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:04.950399 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 171228us
I20251023 02:05:04.922187 21962 raft_consensus.cc:1275] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:05:04.952795 22076 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:04.958370 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:04.895808 23593 consensus_queue.cc:1059] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20251023 02:05:04.962379 22031 raft_consensus.cc:1275] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:04.936311 23891 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:04.967897 22077 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:04.961771 23819 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:04.977378 22034 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:04.974748 23751 raft_consensus.cc:1027] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: attempt to promote peer 1ab8b0b58147446d84d901b4ed2ff39e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20251023 02:05:04.997866 23005 raft_consensus.cc:697] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:04.965718 23743 raft_consensus.cc:740] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:04.955997 23758 consensus_queue.cc:1059] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20251023 02:05:05.040740 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:05.038728 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:05.041440 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 48281us
I20251023 02:05:04.969918 23703 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:04.957098 23762 consensus_queue.cc:1059] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20251023 02:05:04.958986 23757 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:04.979924 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:05.050064 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 97370.7us
I20251023 02:05:04.937309 23869 consensus_queue.cc:237] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.052368 22076 leader_election.cc:400] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5.
I20251023 02:05:04.995363 23473 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:05.047981 21925 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.047490 22075 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:05.051322 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 138155us
I20251023 02:05:05.047782 22029 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:04.976238 23689 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:05.045761 23906 log.cc:927] Time spent T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20251023 02:05:04.992563 23928 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:05.032496 23819 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:05.057691 22076 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:05.045471 22028 raft_consensus.cc:1240] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:05.037135 23727 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:05.037339 23732 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:05.051005 22102 raft_consensus.cc:2393] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:05.055909 23005 consensus_queue.cc:237] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.062497 21925 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.054563 23758 raft_consensus.cc:3055] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:05.085724 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:05.090610 22621 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:05.054280 23920 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:05.091861 22075 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:05.069375 23913 log.cc:927] Time spent T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20251023 02:05:05.094079 22076 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:05.089730 22109 raft_consensus.cc:1240] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:05.061018 23733 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:05.069290 23917 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20251023 02:05:04.970155 23923 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:05:05.098265 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:05.113816 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:05.061724 23819 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:05.121680 23778 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.122241 23778 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.124419 23778 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:05.130388 23593 consensus_queue.cc:1048] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:05:05.131240 23593 consensus_queue.cc:1243] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251023 02:05:05.137143 22034 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:05.141551 22077 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:05.144964 23755 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:05.060799 23520 consensus_queue.cc:1059] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20251023 02:05:05.156000 23891 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:05.075240 23743 consensus_queue.cc:260] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.157711 23743 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:05:05.158831 23407 raft_consensus.cc:1068] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: Unable to promote non-voter 1ab8b0b58147446d84d901b4ed2ff39e: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:05.160801 21961 raft_consensus.cc:1275] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251023 02:05:05.161163 21961 pending_rounds.cc:85] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 2
I20251023 02:05:05.161410 21961 pending_rounds.cc:107] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251023 02:05:05.161664 21961 raft_consensus.cc:2889] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:05.164397 23406 raft_consensus.cc:3055] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:05.164741 23406 raft_consensus.cc:740] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:05.165867 23799 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.100106 21925 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:05.063093 23709 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:05.061161 23642 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.060979 23717 raft_consensus.cc:3055] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
W20251023 02:05:05.070652 23924 log.cc:927] Time spent T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20251023 02:05:05.063104 23732 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:05.109663 23910 log.cc:927] Time spent T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.342s user 0.004s sys 0.000s
I20251023 02:05:05.110249 23842 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:05.166823 23733 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:05.090247 23758 raft_consensus.cc:740] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:05.169366 23703 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.115320 22002 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:05.239332 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 124085us
I20251023 02:05:05.063405 23727 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:05.118528 22030 raft_consensus.cc:2393] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:05.119496 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:05.139495 23930 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.165563 23406 consensus_queue.cc:260] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.058075 23689 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:05.173942 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 79965.7us
I20251023 02:05:05.097752 23928 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:05.068887 23866 consensus_queue.cc:1048] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
W20251023 02:05:05.223914 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 123933us
I20251023 02:05:05.110872 23757 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:05.260763 21960 raft_consensus.cc:1240] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:05.085450 23725 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:05.238576 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 146820us
I20251023 02:05:05.220710 23755 raft_consensus.cc:2749] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:05.220623 23731 raft_consensus.cc:3055] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
W20251023 02:05:05.145305 21858 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33304 (request call id 92) took 11401 ms (client timeout 15000 ms). Trace:
I20251023 02:05:05.224347 23828 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:05.253221 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:05.230389 23732 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:05.254184 21958 raft_consensus.cc:1275] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:05.221859 23717 raft_consensus.cc:740] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:05.247131 23727 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:05.226576 23923 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:05.277827 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:05.183698 23799 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.268121 22029 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:05.278546 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:05.268663 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:05.271258 21961 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251023 02:05:05.279098 21958 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:05:05.274164 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:05.274873 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:05.238126 23733 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:05.246183 23407 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:05.264042 21925 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.285984 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:05.288949 21959 raft_consensus.cc:1240] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20251023 02:05:05.281195 21957 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:05.281711 22076 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:05.294931 22002 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:05.231885 23671 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.294181 21956 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:05.272271 23912 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.302954 22002 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:05.303200 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 161745us
I20251023 02:05:05.267151 23731 raft_consensus.cc:740] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:05.300112 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:05.312144 22002 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:05.297703 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:05.272866 23406 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:05.229719 23642 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:05.276808 23828 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:05.307473 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:05.260365 23779 consensus_queue.cc:1048] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20251023 02:05:05.293502 23733 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:05.324205 22621 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:05.293975 23407 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:05.256485 23687 consensus_queue.cc:1059] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20251023 02:05:05.282371 23713 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:05.273141 22028 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:05.311430 23593 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:05.355698 21928 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:05.329506 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 72818.2us
I20251023 02:05:05.362164 21925 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:05.356941 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 96412.7us
I20251023 02:05:05.363807 22076 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:05.317482 21961 pending_rounds.cc:85] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 1
I20251023 02:05:05.364701 21961 pending_rounds.cc:107] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251023 02:05:05.365010 21961 raft_consensus.cc:2889] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251023 02:05:05.365718 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:05.316859 23911 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:05.326316 23755 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251023 02:05:05.373522 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:05.376567 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:05.323745 23828 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:05.370293 22033 raft_consensus.cc:1275] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:05.297753 23935 consensus_queue.cc:1048] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.057s
W20251023 02:05:05.387486 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:05.392707 21957 raft_consensus.cc:1240] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20251023 02:05:05.393218 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 106995us
I20251023 02:05:05.397090 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:05.398244 22110 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:05:05.399019 21925 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.328819 23407 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:05.257954 23930 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.402381 21925 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.407466 21925 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.296353 23758 consensus_queue.cc:260] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.410770 23758 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:05.370333 23593 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:05.386145 23547 consensus_meta.cc:287] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.543s user 0.009s sys 0.002s
I20251023 02:05:05.369679 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:05.435527 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 87960.3us
I20251023 02:05:05.436231 23687 consensus_queue.cc:1059] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251023 02:05:05.382717 23928 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:05.354158 23842 consensus_queue.cc:1059] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20251023 02:05:05.385507 23755 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:05.422179 23726 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251023 02:05:05.455560 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 56458.7us
I20251023 02:05:05.328835 23934 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.128s
I20251023 02:05:05.400848 23888 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.316080 23912 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:05.416846 23739 consensus_meta.cc:287] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.208s user 0.002s sys 0.012s
I20251023 02:05:05.419157 23774 raft_consensus.cc:2749] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:05.470446 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:05.504592 22033 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:05.455785 23914 log.cc:927] Time spent T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.062s user 0.000s sys 0.003s
I20251023 02:05:05.430939 23869 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.431465 23593 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:05:05.479619 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 115904us
I20251023 02:05:05.447577 23473 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:05.438112 23713 raft_consensus.cc:697] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:05.517624 21956 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:05.507750 22076 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:05.470397 23755 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
W20251023 02:05:05.523056 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 115657us
I20251023 02:05:05.382905 23717 consensus_queue.cc:260] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.525221 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:05.525662 21925 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.525807 21959 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:05.501348 23703 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:05.395079 23911 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:05.450040 23547 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:05.517864 23737 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.514174 23866 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:05.266371 21858 rpcz_store.cc:269] 1023 02:04:53.744034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:53.744292 (+ 258us) service_pool.cc:225] Handling call
1023 02:05:05.145122 (+11400830us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:05:02.071900 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:05:02.351308 (+279408us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:05:02.356765 (+ 5457us) write_op.cc:435] Acquired schema lock
1023 02:05:02.358322 (+ 1557us) tablet.cc:660] Decoding operations
1023 02:05:02.360908 (+ 2586us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:05:02.360968 (+ 60us) write_op.cc:641] Partition lock acquired for write op
1023 02:05:02.360997 (+ 29us) tablet.cc:683] Acquiring locks for 10 operations
1023 02:05:02.362925 (+ 1928us) tablet.cc:699] Row locks acquired
1023 02:05:02.362946 (+ 21us) write_op.cc:260] PREPARE: finished
1023 02:05:02.363091 (+ 145us) write_op.cc:270] Start()
1023 02:05:02.363215 (+ 124us) write_op.cc:276] Timestamp: P: 1761185102363036 usec, L: 0
1023 02:05:02.363235 (+ 20us) op_driver.cc:348] REPLICATION: starting
1023 02:05:02.363979 (+ 744us) log.cc:844] Serialized 4592 byte log entry
1023 02:05:02.634522 (+270543us) op_driver.cc:464] REPLICATION: finished
1023 02:05:02.642011 (+ 7489us) write_op.cc:301] APPLY: starting
1023 02:05:02.642094 (+ 83us) tablet.cc:1365] starting BulkCheckPresence
1023 02:05:02.642240 (+ 146us) tablet.cc:1368] finished BulkCheckPresence
1023 02:05:02.642332 (+ 92us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:05:02.643186 (+ 854us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:05:02.643331 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
1023 02:05:02.643358 (+ 27us) write_op.cc:312] APPLY: finished
1023 02:05:02.643549 (+ 191us) log.cc:844] Serialized 76 byte log entry
1023 02:05:02.643714 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:05:02.643829 (+ 115us) write_op.cc:454] Released schema lock
1023 02:05:02.643877 (+ 48us) write_op.cc:341] FINISH: Updating metrics
1023 02:05:02.644214 (+ 337us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:05:02.644246 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":562206,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":7093,"apply.run_cpu_time_us":2178,"apply.run_wall_time_us":2177,"num_ops":10,"prepare.queue_time_us":255489,"prepare.run_cpu_time_us":15007,"prepare.run_wall_time_us":363856,"raft.queue_time_us":69307,"raft.run_cpu_time_us":4370,"raft.run_wall_time_us":45701,"replication_time_us":241211,"spinlock_wait_cycles":2043392,"thread_start_us":193366,"threads_started":4,"wal-append.queue_time_us":97118,"wal-append.run_cpu_time_us":32855,"wal-append.run_wall_time_us":1182643}]]}
I20251023 02:05:05.614673 21925 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.415603 23731 consensus_queue.cc:260] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.520761 23637 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:05.620432 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 112825us
I20251023 02:05:05.506542 23923 consensus_queue.cc:1048] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20251023 02:05:05.563952 23762 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:05.561921 23901 log.cc:927] Time spent T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
W20251023 02:05:05.628177 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 100289us
I20251023 02:05:05.631058 22076 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:05.622421 23671 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:05.623277 23914 log.cc:927] Time spent T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20251023 02:05:05.638346 21925 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.611578 23473 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:05.618047 22617 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
W20251023 02:05:05.636744 23931 log.cc:927] Time spent T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.245s user 0.000s sys 0.001s
I20251023 02:05:05.617885 23736 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.644915 21925 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.650645 22033 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:05.630448 23737 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:05.651589 23713 consensus_queue.cc:237] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.652484 23911 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:05.656973 23872 consensus_meta.cc:287] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.624s user 0.008s sys 0.000s
I20251023 02:05:05.657737 23569 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:05.662307 23943 log.cc:927] Time spent T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.171s user 0.000s sys 0.000s
I20251023 02:05:05.662786 23912 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:05.514787 23888 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:05.665913 23671 consensus_queue.cc:1059] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:05.669782 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:05.671875 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:05.619169 23775 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:05.520641 23869 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.622941 23921 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:05.673513 21959 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:05.633841 23703 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:05.687322 23743 consensus_meta.cc:287] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.529s user 0.007s sys 0.002s
I20251023 02:05:05.634331 23866 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.688164 23689 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:05.689993 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:05.699617 22035 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:05.636952 23637 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:05.717988 22002 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:05.642776 23762 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.699471 23950 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:05.698091 23935 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.757807 22590 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:05.693094 23937 raft_consensus.cc:3055] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:05.714468 23775 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:05.693795 23473 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:05.696811 23936 raft_consensus.cc:2804] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader election won for term 2
I20251023 02:05:05.692426 23736 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:05.762549 23671 raft_consensus.cc:2804] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:05:05.779747 23642 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:05.695494 23872 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.762780 23845 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:05.763447 23842 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:05.800982 23945 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.801501 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:05.801688 23689 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:05.821607 22621 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
W20251023 02:05:05.806628 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 175671us
I20251023 02:05:05.778856 23937 raft_consensus.cc:740] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:05.709486 23758 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:05.785187 23775 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:05.761346 23703 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:05.810593 23819 consensus_meta.cc:287] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.690s user 0.006s sys 0.005s
W20251023 02:05:05.819801 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 174989us
I20251023 02:05:05.842356 21925 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.843912 21925 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:05.812057 23925 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.005s sys 0.005s
I20251023 02:05:05.820565 22031 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:05.823483 22076 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:05.830804 23739 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:05.776729 23935 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:05.847352 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:05.851557 22076 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:05.818403 23671 raft_consensus.cc:697] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:05.820043 23936 raft_consensus.cc:697] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:05.849481 22030 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:05.862618 22076 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:05:05.857964 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:05.859812 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:05.860666 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:05.840159 23951 log.cc:927] Time spent T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.210s user 0.000s sys 0.001s
I20251023 02:05:05.856637 21925 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.863740 22621 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
W20251023 02:05:05.845929 23949 log.cc:927] Time spent T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.229s user 0.001s sys 0.000s
I20251023 02:05:05.832105 23958 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:05.885561 22076 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:05.886277 21955 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s user 0.005s sys 0.004s
W20251023 02:05:05.854898 23925 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s user 0.005s sys 0.005s
W20251023 02:05:05.885602 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:05.834904 22615 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:05.857115 23737 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:05.956600 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 71119.8us
W20251023 02:05:05.953125 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 96564.9us
I20251023 02:05:05.957103 22001 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:05.849285 23935 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:05.846311 23963 raft_consensus.cc:2749] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:05.862396 23758 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:05.839077 23816 consensus_queue.cc:1059] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20251023 02:05:05.862242 23869 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:05.861879 23473 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:05.821619 23956 consensus_queue.cc:1059] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.249s
I20251023 02:05:05.958789 23642 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:05.959787 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:06.004393 23642 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.004956 23919 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:06.005636 23919 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:06.006143 23919 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:06.007282 23642 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:06.007877 23953 log.cc:927] Time spent T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.195s user 0.005s sys 0.005s
W20251023 02:05:05.972080 23828 consensus_meta.cc:287] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.533s user 0.009s sys 0.000s
I20251023 02:05:06.009052 23828 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.009711 23297 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:05.833050 23945 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:05:05.959386 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1058) took 1398 ms. Trace:
I20251023 02:05:05.854108 23819 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.847290 23739 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.886912 23911 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:05.957758 23406 consensus_meta.cc:287] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.631s user 0.009s sys 0.005s
W20251023 02:05:05.958498 23907 log.cc:927] Time spent T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 1.467s user 0.004s sys 0.000s
I20251023 02:05:06.001135 23969 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:05.962085 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:05.969815 23964 raft_consensus.cc:2749] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:06.005250 23775 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:05.998210 21925 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:05.847378 23937 consensus_queue.cc:260] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.018744 21960 rpcz_store.cc:276] 1023 02:05:04.560728 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:04.913386 (+352658us) service_pool.cc:225] Handling call
1023 02:05:04.913783 (+ 397us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:05.261294 (+347511us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:05.959145 (+697851us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b5400302354. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:05.959330 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":797466496}
I20251023 02:05:06.034183 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.036782 23944 consensus_queue.cc:1048] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.066s
I20251023 02:05:06.037076 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.037680 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:05.962198 23950 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:06.010562 23942 log.cc:927] Time spent T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.340s user 0.001s sys 0.000s
I20251023 02:05:06.038604 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:05.992484 23758 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:05.995203 23957 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:06.026016 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:05.999845 23547 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:06.012308 23762 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:05.997784 22076 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:06.032552 23947 log.cc:927] Time spent T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.259s user 0.000s sys 0.001s
I20251023 02:05:06.062197 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:06.062676 21925 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:06.062844 22622 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:06.063164 23965 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:06.063437 23975 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:06.024071 23814 raft_consensus.cc:3055] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:06.022315 23816 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:06.038714 23950 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.036962 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:06.019441 21928 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.060s sys 0.045s
I20251023 02:05:06.040939 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:05.971297 23737 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:06.041774 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.042001 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.048219 22036 raft_consensus.cc:1240] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:06.048439 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:06.031077 23946 log.cc:927] Time spent T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.245s user 0.001s sys 0.000s
I20251023 02:05:06.115921 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:06.059448 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.117936 22105 raft_consensus.cc:2410] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:05:06.033430 23963 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:06.113294 23737 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:06.040409 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
W20251023 02:05:06.057962 23955 log.cc:927] Time spent T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.222s user 0.000s sys 0.001s
I20251023 02:05:06.027403 23911 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.064548 21925 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:06.047389 23758 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:06.124429 23775 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.018007 23671 consensus_queue.cc:237] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:06.119875 23593 consensus_meta.cc:287] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.595s user 0.008s sys 0.001s
I20251023 02:05:06.063594 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.067514 22001 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.051250 23842 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:06.046833 23969 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:05.971470 23930 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.040831 23736 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:06.048390 23925 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:06.065550 23814 raft_consensus.cc:740] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:06.055389 23957 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.137187 23739 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.035055 23473 consensus_queue.cc:1048] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.025s
W20251023 02:05:06.069363 23959 log.cc:927] Time spent T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.229s user 0.001s sys 0.000s
W20251023 02:05:06.069242 23972 log.cc:927] Time spent T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20251023 02:05:06.072453 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1342) took 1533 ms. Trace:
I20251023 02:05:06.111828 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1072) took 1235 ms. Trace:
I20251023 02:05:06.067834 23888 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:06.112428 21928 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.061s sys 0.046s
I20251023 02:05:06.072909 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:06.073064 21953 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251023 02:05:06.073139 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.114938 22035 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:06.113914 21960 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.115324 22110 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
W20251023 02:05:06.032019 23725 consensus_meta.cc:287] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.765s user 0.008s sys 0.003s
I20251023 02:05:06.113610 22104 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
W20251023 02:05:06.177059 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 718192us
I20251023 02:05:06.068514 23968 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:06.101760 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 104080us
W20251023 02:05:06.114274 23407 consensus_meta.cc:287] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.682s user 0.000s sys 0.009s
W20251023 02:05:06.066642 23962 log.cc:927] Time spent T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.235s user 0.000s sys 0.001s
I20251023 02:05:06.182154 23725 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:06.182840 23407 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:06.183913 22076 connection.cc:232] RPC call timeout handler was delayed by -1.2310889100000395s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20870000000000002s for a response
I20251023 02:05:06.184841 23407 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:06.185029 23954 log.cc:927] Time spent T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.668s user 0.002s sys 0.000s
I20251023 02:05:06.185567 22109 raft_consensus.cc:1240] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:06.174223 23814 consensus_queue.cc:260] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.186489 23814 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.187435 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1411) took 1089 ms. Trace:
I20251023 02:05:06.187593 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.187778 22109 rpcz_store.cc:276] 1023 02:05:05.097972 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:05.099329 (+ 1357us) service_pool.cc:225] Handling call
1023 02:05:05.118062 (+ 18733us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:06.185985 (+1067923us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:06.187253 (+ 1268us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001ab3d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:06.187388 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2454804352}
I20251023 02:05:06.188194 21962 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:06.189033 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.189556 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.190600 23950 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:06.191079 22002 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.192052 23957 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.192575 23930 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:06.192955 23930 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:06.193279 23930 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.141024 23945 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:06.195297 23920 consensus_meta.cc:287] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.092s user 0.010s sys 0.000s
I20251023 02:05:06.195581 23920 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:06.168427 23819 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.140784 23925 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:06.071192 23727 consensus_meta.cc:287] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.771s user 0.006s sys 0.003s
I20251023 02:05:06.072582 23762 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election won for term 4
W20251023 02:05:06.123327 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 58855.6us
W20251023 02:05:06.069401 23755 consensus_meta.cc:287] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.507s user 0.008s sys 0.000s
I20251023 02:05:06.115465 23937 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:05:06.120007 23970 log.cc:927] Time spent T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.230s user 0.001s sys 0.000s
I20251023 02:05:06.066074 23961 raft_consensus.cc:2749] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:06.047811 23872 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:06.069629 23726 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:06.067015 23828 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:06.129451 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 62018.2us
I20251023 02:05:06.073627 23703 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:06.122567 23737 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:06.130945 23971 log.cc:927] Time spent T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20251023 02:05:06.125931 22615 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:06.133801 23733 consensus_meta.cc:287] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.775s user 0.006s sys 0.004s
I20251023 02:05:06.225859 23733 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.215911 23726 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.228482 23733 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:06.134119 23967 log.cc:927] Time spent T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
I20251023 02:05:06.216101 23762 raft_consensus.cc:697] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:06.125265 23963 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:06.072530 23816 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.056658 23944 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:06.131824 23775 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:06.237622 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:06.237802 23775 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:06.238265 23775 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:06.239084 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:06.174155 22037 rpcz_store.cc:276] 1023 02:05:04.876260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:04.876395 (+ 135us) service_pool.cc:225] Handling call
1023 02:05:04.876719 (+ 324us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:05.246881 (+370162us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:05.246911 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:05.272182 (+ 25271us) log.cc:844] Serialized 26 byte log entry
1023 02:05:05.412645 (+140463us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:05.412684 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:06.073178 (+660494us) spinlock_profiling.cc:243] Waited 40.2 ms on lock 0x7b54002e9354. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:06.073223 (+ 45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:06.073267 (+ 44us) raft_consensus.cc:1685] finished
1023 02:05:06.073283 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:06.111745 (+ 38462us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28867,"raft.queue_time_us":39439,"raft.run_cpu_time_us":373,"raft.run_wall_time_us":73825,"spinlock_wait_cycles":92402432,"thread_start_us":390565,"threads_started":2,"wal-append.queue_time_us":392150}
I20251023 02:05:06.242214 23762 consensus_queue.cc:237] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.130280 23593 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.246606 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.246773 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.138448 23911 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.179680 21928 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.248746 22104 raft_consensus.cc:2393] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
W20251023 02:05:06.141330 23732 consensus_meta.cc:287] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.849s user 0.010s sys 0.000s
W20251023 02:05:06.249027 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 69371.5us
I20251023 02:05:06.250355 21928 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:06.168761 23964 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:06.252919 23964 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.201527 21925 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:06.174836 21954 rpcz_store.cc:276] 1023 02:05:04.538789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:04.883945 (+345156us) service_pool.cc:225] Handling call
1023 02:05:04.898402 (+ 14457us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:05.166439 (+268037us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:05.296071 (+129632us) log.cc:844] Serialized 14 byte log entry
1023 02:05:05.466374 (+170303us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:05.466442 (+ 68us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:05.466503 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:06.072205 (+605702us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b5400218c54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:06.072258 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:06.072399 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":264697,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":5440,"raft.queue_time_us":70472,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":4773,"spinlock_wait_cycles":616034688,"thread_start_us":299285,"threads_started":3,"wal-append.queue_time_us":224475}
I20251023 02:05:06.207746 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.202553 23727 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.209291 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.138046 23842 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.262148 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.262715 21959 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.217012 22105 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251023 02:05:06.212529 23925 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.034389 23936 consensus_queue.cc:237] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.230803 21960 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:06.234503 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:06.235263 22033 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251023 02:05:06.264290 22002 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.247385 21956 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.237325 23944 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.252063 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:06.252368 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:06.217036 23758 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.266710 22034 raft_consensus.cc:2393] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:06.138543 23774 raft_consensus.cc:3055] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:06.210790 23945 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:06.268056 23774 raft_consensus.cc:740] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:06.256599 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 55161.7us
I20251023 02:05:06.257386 21962 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:06.200484 23968 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.258757 23473 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:06.259393 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.300781 22104 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:06.215289 22035 raft_consensus.cc:1275] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:06.269138 23758 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:06.206789 23755 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.302023 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.267481 23407 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:06.302614 22105 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:06.302863 23407 raft_consensus.cc:697] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:06.303581 23755 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:06.304296 22110 raft_consensus.cc:1240] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:06.140290 23969 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.305758 21961 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:06.306295 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:06.306807 23968 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.307200 23969 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.296553 23925 consensus_queue.cc:1059] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20251023 02:05:06.308699 23731 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:06.310340 23816 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.313196 23593 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.315037 23717 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:06.319676 21955 raft_consensus.cc:1275] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 2. (index mismatch)
I20251023 02:05:06.269503 22106 raft_consensus.cc:2393] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:06.278196 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.293766 21925 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:06.322510 22109 raft_consensus.cc:1240] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
W20251023 02:05:06.278033 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:06.322661 21925 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:06.324048 21925 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:06.303834 23407 consensus_queue.cc:237] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:06.319756 23799 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.324065 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:06.255643 23828 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:06.332442 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.333000 21955 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:06.268723 23964 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:06.324802 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:06.324990 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:06.324967 21954 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.295596 21960 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:06.268899 23775 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.330179 21959 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:06.336906 21928 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:06.331326 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:06.262657 23963 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:05:06.331714 21925 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:06.327147 21956 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:06.272130 23866 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:06.338161 22034 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:06.338928 22002 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.351655 22002 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.279821 23727 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:06.269812 23944 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:06.336596 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.336732 23919 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.333958 23755 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:06.281721 23842 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:06.346016 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:06.369033 22590 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.369185 21955 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:06.370311 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.256358 23732 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.334921 23799 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:06.331669 23845 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:06.353981 21960 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:06.341769 21956 raft_consensus.cc:1240] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20251023 02:05:06.326660 23593 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:06.355707 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.356220 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.402614 22104 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:06.357272 22615 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:06.357486 23919 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.358757 23737 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.339828 23473 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:06.360378 23726 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.361135 23814 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:06.361719 23671 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:06.349861 21928 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.366250 22105 raft_consensus.cc:1275] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251023 02:05:06.366990 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.426330 22105 pending_rounds.cc:85] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 2
I20251023 02:05:06.426627 22105 pending_rounds.cc:107] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251023 02:05:06.426834 22105 raft_consensus.cc:2889] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:06.426657 22106 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:06.428057 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.367413 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:06.370430 22002 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.347206 21962 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:06.429453 22031 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:06.430102 22621 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:06.430186 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.430914 21962 raft_consensus.cc:2393] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:06.431695 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.432363 21962 raft_consensus.cc:2393] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:06.428064 21925 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:06.372753 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.352885 23925 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:06.378114 23911 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:06.341007 23828 raft_consensus.cc:697] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:06.372020 23755 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.382061 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.383553 23866 consensus_queue.cc:1048] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:06.353188 23969 consensus_queue.cc:1048] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20251023 02:05:06.385779 23964 consensus_queue.cc:1048] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:06.452893 23732 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:06.389448 23775 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:06.390439 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.414450 23758 consensus_queue.cc:1059] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20251023 02:05:06.454409 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.455566 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.435111 23739 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:05:06.456254 21959 raft_consensus.cc:2393] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:06.434517 23547 consensus_queue.cc:1059] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20251023 02:05:06.457005 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.335493 23816 consensus_queue.cc:1048] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20251023 02:05:06.391621 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:06.458680 23828 consensus_queue.cc:237] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.423029 23717 consensus_queue.cc:1059] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20251023 02:05:06.460557 23717 raft_consensus.cc:3055] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:06.460996 23717 raft_consensus.cc:740] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:06.394335 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1082) took 1090 ms. Trace:
I20251023 02:05:06.394816 23779 raft_consensus.cc:3055] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:06.462323 23779 raft_consensus.cc:740] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:06.397109 23637 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:06.444667 23911 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:06.463186 22108 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:06.371115 23774 consensus_queue.cc:260] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.462065 23717 consensus_queue.cc:260] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.464320 23774 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:06.444813 23842 consensus_queue.cc:1048] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20251023 02:05:06.464882 23717 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.467481 23737 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:06.447283 23703 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:06.468341 23703 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:06.461719 22036 rpcz_store.cc:276] 1023 02:05:05.304197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:05.304487 (+ 290us) service_pool.cc:225] Handling call
1023 02:05:05.326295 (+ 21808us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:06.114554 (+788259us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:06.394108 (+279554us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b540001d1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:06.394282 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1659777408}
I20251023 02:05:06.414672 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.468868 23703 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:06.422796 23593 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.413807 22076 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:06.415169 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.422034 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.327591 23945 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.476336 22029 raft_consensus.cc:2410] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251023 02:05:06.429260 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:06.435473 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 65128.7us
W20251023 02:05:06.435606 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 85829.3us
I20251023 02:05:06.476984 21956 raft_consensus.cc:2393] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:06.433084 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.421957 23818 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:06.422619 23963 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:05:06.425298 23473 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:06.450786 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.535734 22105 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:06.425000 23297 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:06.463021 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.536880 21961 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:06.463529 23911 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:06.473366 22104 raft_consensus.cc:2393] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:06.470649 21859 catalog_manager.cc:5649] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 2 to 4, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:06.469326 21960 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:06.461228 23755 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:06.397043 21954 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:06.477030 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1357) took 1221 ms. Trace:
W20251023 02:05:06.478295 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 64626.2us
I20251023 02:05:06.527767 22002 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.445310 23814 raft_consensus.cc:697] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:06.479403 21908 placement_policy_util.cc:244] tablet 5e923b66ccf04dd8985bbfe31adbf0b4: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
I20251023 02:05:06.422811 23944 consensus_queue.cc:1048] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
I20251023 02:05:06.445046 23671 raft_consensus.cc:697] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:06.472081 23941 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.011s sys 0.034s
I20251023 02:05:06.434695 23923 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251023 02:05:06.504716 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:06.548679 22076 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:06.541513 21958 rpcz_store.cc:276] 1023 02:05:05.255819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:05.292696 (+ 36877us) service_pool.cc:225] Handling call
1023 02:05:05.293012 (+ 316us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:05.323845 (+ 30833us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:05.323873 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:05.355991 (+ 32118us) log.cc:844] Serialized 52 byte log entry
1023 02:05:05.680706 (+324715us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:05.719670 (+ 38964us) log.cc:844] Serialized 14 byte log entry
1023 02:05:06.362770 (+643100us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x7b1800344980. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9cdd37 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9e8445 00007f9c5fa90c00 00007f9c5faa32a6 00007f9c5fab3e17 00007f9c5fab3d98
1023 02:05:06.445477 (+ 82707us) log.cc:844] Serialized 14 byte log entry
1023 02:05:06.445717 (+ 240us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:06.445787 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:06.476829 (+ 31042us) raft_consensus.cc:1685] finished
1023 02:05:06.476846 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:06.476980 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15518,"prepare.queue_time_us":38882,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":3681,"raft.queue_time_us":115687,"raft.run_cpu_time_us":447,"raft.run_wall_time_us":33857,"spinlock_wait_cycles":11770624,"thread_start_us":331579,"threads_started":2,"wal-append.queue_time_us":380285}
I20251023 02:05:06.560154 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.561023 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1349) took 1309 ms. Trace:
I20251023 02:05:06.563199 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.561388 21953 rpcz_store.cc:276] 1023 02:05:05.251751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:05.251867 (+ 116us) service_pool.cc:225] Handling call
1023 02:05:06.072708 (+820841us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:06.176327 (+103619us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:06.560835 (+384508us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b5400218c50. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:06.560971 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1837245312}
I20251023 02:05:06.567121 21859 catalog_manager.cc:5649] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 4, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:05:06.480878 23739 raft_consensus.cc:697] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:06.576228 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:06.576741 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:06.578126 21960 raft_consensus.cc:2393] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:06.580210 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.581523 21960 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 2.
I20251023 02:05:06.537914 23785 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:06.584069 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.584596 22105 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 7
I20251023 02:05:06.584398 23785 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.570559 23297 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.585973 22589 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:06.529829 23866 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:06.586894 21961 raft_consensus.cc:1240] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:06.587353 21908 placement_policy_util.cc:244] tablet 6dddaeefc3d449a0ba9b3e6695e37507: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
W20251023 02:05:06.588209 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 60722.6us
I20251023 02:05:06.591518 22077 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:06.558535 23941 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s user 0.011s sys 0.034s
I20251023 02:05:06.602645 21859 catalog_manager.cc:5649] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 3, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:06.601184 22002 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.602152 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.611363 21956 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:06.538414 23473 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:06.600176 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.558859 23923 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:06.538329 23818 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.585527 23888 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
W20251023 02:05:06.612730 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 64164.6us
I20251023 02:05:06.643409 22076 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:06.616321 22002 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.619796 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.623708 23736 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:06.647240 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:06.623915 23785 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.584884 23671 consensus_queue.cc:237] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.626066 23297 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:06.626382 23945 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:06.627296 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:06.635006 22000 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:06.636282 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.570919 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
W20251023 02:05:06.641103 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 55409.9us
I20251023 02:05:06.644666 21960 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:06.639770 23923 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.648048 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.609342 23547 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:06.651382 22002 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:06.651918 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.652985 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.678427 22103 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:06.653529 23979 log.cc:927] Time spent T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
W20251023 02:05:06.656934 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 79909.1us
I20251023 02:05:06.612958 23637 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:06.719033 22590 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:06.722234 23985 log.cc:927] Time spent T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20251023 02:05:06.722124 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:06.666720 21954 raft_consensus.cc:1240] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
W20251023 02:05:06.723146 23978 log.cc:927] Time spent T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.196s user 0.000s sys 0.001s
W20251023 02:05:06.724833 23977 log.cc:927] Time spent T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.209s user 0.008s sys 0.004s
I20251023 02:05:06.726166 22618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 280) took 1417 ms. Trace:
I20251023 02:05:06.726538 22618 rpcz_store.cc:276] 1023 02:05:05.308779 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:05.315163 (+ 6384us) service_pool.cc:225] Handling call
1023 02:05:05.478297 (+163134us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:05.589620 (+111323us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:05.616933 (+ 27313us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:06.124470 (+507537us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b2c0004b768. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5e3949b1 00007f9c5e3a3fe2 00007f9c5fad4701 00007f9c5fa98fda 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22
1023 02:05:06.175242 (+ 50772us) log.cc:844] Serialized 548 byte log entry
1023 02:05:06.335891 (+160649us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:06.371489 (+ 35598us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:06.414245 (+ 42756us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:06.725877 (+311632us) raft_consensus.cc:1685] finished
1023 02:05:06.725904 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:06.726109 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25932672,"thread_start_us":131701,"threads_started":1,"wal-append.queue_time_us":253463}
I20251023 02:05:06.728463 23732 raft_consensus.cc:3055] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:05:06.728847 23732 raft_consensus.cc:740] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:06.731719 23637 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:06.732826 21954 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20251023 02:05:06.734295 23919 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:06.667585 23407 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:06.668298 23956 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:06.670367 22104 raft_consensus.cc:1240] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:06.670608 21960 raft_consensus.cc:1275] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:05:06.645903 23930 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.645319 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:06.743338 21961 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:06.672765 21953 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:06.673453 21928 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:06.652169 23713 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.015s sys 0.012s
W20251023 02:05:06.677352 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 91454us
I20251023 02:05:06.677783 22615 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 5
I20251023 02:05:06.739759 21926 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:06.744961 22589 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:06.655362 22106 raft_consensus.cc:1275] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 2. (term mismatch)
I20251023 02:05:06.746992 22106 pending_rounds.cc:85] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 1
I20251023 02:05:06.747334 22106 pending_rounds.cc:107] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251023 02:05:06.747617 22106 raft_consensus.cc:2889] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:06.748003 22106 pending_rounds.cc:107] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251023 02:05:06.748276 22106 raft_consensus.cc:2889] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:06.690517 22105 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:06.644599 23593 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.701519 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.755095 22108 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.650949 23721 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:06.647342 23736 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:06.757736 23736 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:06.749377 23828 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:05:06.705403 23941 raft_consensus.cc:2749] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:06.712661 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.772828 22037 raft_consensus.cc:2393] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
W20251023 02:05:06.715360 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:06.642951 23936 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:06.736300 23956 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.785625 23936 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.787364 23956 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.651197 23937 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:06.671517 23520 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:06.668808 23742 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:06.626086 23726 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.741127 21859 catalog_manager.cc:5649] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:06.677995 23547 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:06.750294 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.823235 22105 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:06.752476 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.745553 23732 consensus_queue.cc:260] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:06.754846 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 85314.2us
I20251023 02:05:06.757210 23721 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:06.710597 23739 consensus_queue.cc:237] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.774259 22109 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:06.749485 23816 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
W20251023 02:05:06.714201 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 70858.1us
I20251023 02:05:06.862005 21925 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:06.862421 22076 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:06.778721 23930 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.781709 23297 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:06.784858 23941 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:06.788266 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:06.750802 23713 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s user 0.016s sys 0.013s
W20251023 02:05:06.798377 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 212871us
I20251023 02:05:06.876608 22103 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:06.877956 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.803907 22031 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:06.707507 23944 consensus_queue.cc:1059] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20251023 02:05:06.799142 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.880132 22104 raft_consensus.cc:2393] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:06.825999 23937 raft_consensus.cc:697] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:06.676878 23814 consensus_queue.cc:237] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.824203 22106 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:06.833846 23990 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:06.834298 23928 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:06.840147 21959 raft_consensus.cc:2410] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20251023 02:05:06.842351 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:06.828341 23547 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:06.825091 23520 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:06.847892 23984 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20251023 02:05:06.848644 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.902148 22105 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:06.896711 22587 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:06.849406 21859 catalog_manager.cc:5649] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:05:06.797837 23936 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:06.856426 23473 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.857750 23732 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:06.863567 21925 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:06.866714 22108 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:06.826913 23742 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.873744 23941 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.875571 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.870019 22111 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:06.919822 21859 catalog_manager.cc:5649] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 3 to 4, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:05:06.919826 22109 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:06.889041 21961 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:06.892105 22076 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:06.897125 22107 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:06.863065 23721 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:06.928719 21962 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
W20251023 02:05:06.898070 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:06.899984 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.931692 22106 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:06.902541 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:06.904812 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:06.790759 23407 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.901881 23965 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:06.936412 23965 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.936709 23911 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.938419 23965 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:06.909247 23937 consensus_queue.cc:237] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.938356 21859 catalog_manager.cc:5649] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 3, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:06.940660 21859 catalog_manager.cc:5649] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 3, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 3 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:06.881142 23755 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:06.894210 23888 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.022s
I20251023 02:05:06.921633 23923 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.942763 23888 consensus_queue.cc:1059] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251023 02:05:06.943512 23888 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:06.943984 23923 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.944684 23888 raft_consensus.cc:3055] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:05:06.914796 23779 consensus_queue.cc:260] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.945050 23888 raft_consensus.cc:740] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:06.900277 23713 raft_consensus.cc:3055] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:06.945542 23779 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.945742 23713 raft_consensus.cc:740] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:06.946610 23713 consensus_queue.cc:260] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.946930 23473 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:06.947744 23713 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.949167 23736 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.910773 23818 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.911773 23928 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.950384 23911 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.951642 23736 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.951936 23928 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:06.912379 23799 consensus_meta.cc:287] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.501s user 0.009s sys 0.000s
I20251023 02:05:06.923336 23990 consensus_queue.cc:1059] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20251023 02:05:06.879832 23816 raft_consensus.cc:697] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:06.953480 23990 raft_consensus.cc:3055] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:05:06.953964 23990 raft_consensus.cc:740] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:06.954255 23816 consensus_queue.cc:237] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:06.912998 23956 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:06.913794 23520 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.957528 23941 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:06.916793 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 53303.3us
I20251023 02:05:06.958029 23956 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.959800 23956 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.960713 22109 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:06.961853 21859 catalog_manager.cc:5649] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:06.962630 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:06.963313 22037 raft_consensus.cc:2393] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:06.963176 23964 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:06.963551 21859 catalog_manager.cc:5649] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 reported cstate change: term changed from 2 to 4, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:05:06.964218 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:06.926844 22076 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:06.966663 23520 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:06.966847 23742 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:06.967151 23936 consensus_queue.cc:1059] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:05:06.952561 23799 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:06.875022 23828 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:06.914016 23703 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.963856 23964 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.967137 23520 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:06.918972 23297 raft_consensus.cc:2804] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:06.934437 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1362) took 1451 ms. Trace:
I20251023 02:05:06.966804 23941 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:06.935834 22618 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
W20251023 02:05:06.978994 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 327704us
I20251023 02:05:06.982384 21957 rpcz_store.cc:276] 1023 02:05:05.483010 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:05.483229 (+ 219us) service_pool.cc:225] Handling call
1023 02:05:05.514675 (+ 31446us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:05.527834 (+ 13159us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:05.527866 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:05.612410 (+ 84544us) log.cc:844] Serialized 26 byte log entry
1023 02:05:06.383513 (+771103us) spinlock_profiling.cc:243] Waited 1.79 ms on lock 0x7b1800021580. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9cdd37 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c5f62f2bb 00007f9c5f634dc3 00007f9c5f635171 00007f9c5fa5465f
1023 02:05:06.532682 (+149169us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:06.618069 (+ 85387us) log.cc:844] Serialized 14 byte log entry
1023 02:05:06.620276 (+ 2207us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:06.900736 (+280460us) spinlock_profiling.cc:243] Waited 6.72 ms on lock 0x7b1800021580. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9b93c6 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9
1023 02:05:06.934009 (+ 33273us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b5400024754. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:06.934074 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:06.934169 (+ 95us) raft_consensus.cc:1685] finished
1023 02:05:06.934189 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:06.934368 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22181,"prepare.queue_time_us":66266,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":17014,"raft.queue_time_us":89660,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":18322,"spinlock_wait_cycles":49118848,"thread_start_us":149024,"threads_started":1,"wal-append.queue_time_us":1018436}
I20251023 02:05:06.986548 22075 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:06.974712 23758 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:06.990566 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:06.994624 22105 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:06.972103 23717 raft_consensus.cc:2804] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:06.998543 21958 raft_consensus.cc:2410] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251023 02:05:07.001219 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:07.004292 21960 raft_consensus.cc:1240] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251023 02:05:07.004416 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:07.005156 22105 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:06.982251 23944 raft_consensus.cc:3055] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:06.974788 23828 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:07.010550 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.011323 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:06.990069 23758 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.014727 22104 raft_consensus.cc:1275] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:07.016306 21960 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:07.020519 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.013037 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.017865 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:07.022034 22103 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:06.954936 23990 consensus_queue.cc:260] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.023433 21958 raft_consensus.cc:2393] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:07.023612 21960 raft_consensus.cc:1240] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:06.984804 23297 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:06.996876 23965 consensus_queue.cc:1048] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.027s
I20251023 02:05:07.008024 22617 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 7: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:06.986104 23941 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:06.986655 22615 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:06.990571 23755 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:07.030009 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.035516 22103 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:07.002826 23717 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:07.008709 23945 raft_consensus.cc:2764] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20251023 02:05:07.054191 23717 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:07.042294 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.016266 23956 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20251023 02:05:07.055934 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.056469 21958 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:07.038635 23919 raft_consensus.cc:2764] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251023 02:05:07.037026 23941 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:07.059397 23758 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:07.061146 23520 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.046147 23990 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:07.062232 22105 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:07.065223 21925 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:07.057325 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.066298 23774 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:07.066742 23774 raft_consensus.cc:697] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:07.066969 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:07.067485 21958 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:07.067538 23774 consensus_queue.cc:237] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.045084 23297 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:07.046762 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:07.076040 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.076717 21959 raft_consensus.cc:2393] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:07.078246 22031 raft_consensus.cc:1275] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:07.047477 23964 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:07.080329 23473 consensus_queue.cc:1048] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:07.052584 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:07.015892 23944 raft_consensus.cc:740] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:07.082955 23944 consensus_queue.cc:260] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.084982 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.054801 23407 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.017760 23828 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:07.089814 22002 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:07.085655 22105 raft_consensus.cc:1275] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20251023 02:05:07.092170 22105 pending_rounds.cc:85] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 2
I20251023 02:05:07.092486 22105 pending_rounds.cc:107] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251023 02:05:07.092773 22105 raft_consensus.cc:2889] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:07.095114 22001 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:07.097259 23941 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.097939 23944 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:07.099650 23941 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:07.102272 23717 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.104394 23736 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:07.104889 23736 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:07.105296 23736 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:07.108855 23969 consensus_queue.cc:1048] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:05:07.085834 21960 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:07.111868 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:07.112605 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:07.112532 21960 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:07.077389 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.114180 21959 raft_consensus.cc:2393] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:07.041921 23928 consensus_queue.cc:1048] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20251023 02:05:07.025368 23888 consensus_queue.cc:260] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.119848 21959 raft_consensus.cc:1275] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251023 02:05:07.123243 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:07.123862 22002 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:07.131209 22030 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:07.131279 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.118150 23888 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:07.158232 21957 raft_consensus.cc:2393] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:07.115309 23818 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:07.042966 23755 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.139875 23407 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:07.168251 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.165998 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:07.170542 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.175251 21925 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:07.176406 21956 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.015s sys 0.013s
W20251023 02:05:07.176810 21956 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.501s user 0.015s sys 0.013s
W20251023 02:05:07.177713 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:07.180433 22002 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:07.159794 23969 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:07.159600 23717 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:07.187153 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 355464us
W20251023 02:05:07.228752 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 482136us
I20251023 02:05:07.243191 22590 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:07.166872 23818 raft_consensus.cc:697] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:07.248349 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.248071 21959 raft_consensus.cc:1240] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20251023 02:05:07.168504 23407 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:07.251336 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.254654 21958 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:07.192466 23473 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:07.301884 23473 raft_consensus.cc:697] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:07.302863 23473 consensus_queue.cc:237] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.279698 23407 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:07.306514 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:07.281384 23991 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:07.308014 23991 raft_consensus.cc:697] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:07.308830 23991 consensus_queue.cc:237] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.241797 23999 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:07.310866 23999 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:07.311236 23999 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:07.283257 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:07.284183 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:07.312603 22032 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:07.312659 22104 raft_consensus.cc:2393] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:07.285722 23928 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:07.286007 23755 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:07.290892 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:07.318308 22106 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:07.244529 23703 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.294220 23818 consensus_queue.cc:237] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.294476 23297 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.314848 22028 raft_consensus.cc:2368] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7.
I20251023 02:05:07.320977 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:07.285902 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:07.325412 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 62129.6us
I20251023 02:05:07.331162 22589 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:07.195873 23969 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.320403 23742 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:07.338680 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.335588 22002 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:07.342981 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.339272 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:07.344475 22107 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:07.349944 22104 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20251023 02:05:07.352947 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:07.321655 23928 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.356212 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.358306 21954 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:07.355993 23755 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:07.356365 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:07.360173 21957 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:07.361979 21954 raft_consensus.cc:1275] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:07.362638 22076 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:07.363866 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.364760 22076 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:07.358858 23755 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.366412 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.329356 23996 consensus_queue.cc:1048] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
W20251023 02:05:07.375056 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:07.374835 21926 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:07.374894 21957 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:07.380833 23757 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:07.381060 23992 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:07.381381 23757 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.381752 23297 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.382176 22076 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:07.381695 23992 consensus_queue.cc:1059] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:05:07.383378 23755 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:07.383380 23757 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:07.383908 23941 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:07.384141 23757 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:07.384382 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:07.384594 23757 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:07.385102 22107 raft_consensus.cc:2368] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4.
I20251023 02:05:07.384596 23964 consensus_queue.cc:1059] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:05:07.386343 21925 leader_election.cc:400] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4.
I20251023 02:05:07.386489 23964 raft_consensus.cc:3055] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:05:07.356968 23737 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:07.385135 23757 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:07.384750 23936 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:07.387266 23737 raft_consensus.cc:697] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:07.387899 23936 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:07.388321 23936 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:07.388598 23737 consensus_queue.cc:237] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.386678 21925 leader_election.cc:403] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20251023 02:05:07.383522 22076 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:07.386169 23780 consensus_queue.cc:1059] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:05:07.394311 21958 raft_consensus.cc:2393] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:07.384977 23755 consensus_queue.cc:1048] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:07.348207 23717 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:07.400491 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.403959 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:07.396447 23941 raft_consensus.cc:697] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:07.404755 23297 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4.
I20251023 02:05:07.390254 23964 raft_consensus.cc:740] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:07.409649 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:07.346041 23742 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.411682 21954 pending_rounds.cc:85] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 2
I20251023 02:05:07.607147 21954 pending_rounds.cc:107] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251023 02:05:07.607503 21954 raft_consensus.cc:2889] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:07.405236 23965 raft_consensus.cc:2804] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:07.397312 23845 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:07.402437 23911 raft_consensus.cc:3055] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:07.421155 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.405097 24003 consensus_queue.cc:1059] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20251023 02:05:07.440099 23944 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:07.453811 23736 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:07.454975 23713 consensus_meta.cc:287] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.507s user 0.011s sys 0.000s
I20251023 02:05:07.459442 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.463501 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251023 02:05:07.466123 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:07.623164 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:07.623217 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:07.463524 23547 consensus_meta.cc:287] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.545s user 0.005s sys 0.005s
I20251023 02:05:07.483359 23739 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:07.487322 23993 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:07.489842 24008 log.cc:927] Time spent T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20251023 02:05:07.490651 23969 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.491964 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:07.629170 22107 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:07.629395 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1084) took 1084 ms. Trace:
I20251023 02:05:07.368211 23721 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.630861 23736 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.629814 21953 rpcz_store.cc:276] 1023 02:05:06.544734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:06.756528 (+211794us) service_pool.cc:225] Handling call
1023 02:05:06.814169 (+ 57641us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:06.814476 (+ 307us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:06.814512 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:06.815114 (+ 602us) log.cc:844] Serialized 26 byte log entry
1023 02:05:07.326339 (+511225us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:07.348205 (+ 21866us) log.cc:844] Serialized 14 byte log entry
1023 02:05:07.373984 (+ 25779us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:07.374083 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:07.474903 (+100820us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b180039f8e0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9b93c6 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9eb4ff 00007f9c5c9ed7a1
1023 02:05:07.482689 (+ 7786us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b180039f8e0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9eb4ff 00007f9c5c9ed7a1 00007f9c5c9ed759 00007f9c5c9ed6e9 00007f9c5c9ed6b1 00007f9c5c9ec9ad 0000000000379254 0000000000379099 00007f9c5c9ce75a
1023 02:05:07.629168 (+146479us) raft_consensus.cc:1685] finished
1023 02:05:07.629201 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:07.629343 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56138,"prepare.queue_time_us":30304,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":27094,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":11626,"spinlock_wait_cycles":16717440,"thread_start_us":436879,"threads_started":1,"wal-append.queue_time_us":724888}
I20251023 02:05:07.626084 24006 raft_consensus.cc:3055] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:07.631637 24006 raft_consensus.cc:740] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:07.632104 23721 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:07.503172 23828 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.632438 24006 consensus_queue.cc:260] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.506001 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:07.634632 22104 raft_consensus.cc:2410] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20251023 02:05:07.634935 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 284349us
I20251023 02:05:07.505445 23928 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:07.530817 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:07.534010 23407 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:07.561254 21962 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.025s sys 0.036s
W20251023 02:05:07.640347 21962 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.025s sys 0.036s
I20251023 02:05:07.640465 21960 raft_consensus.cc:1240] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:07.569324 23936 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.395780 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:07.576685 23990 consensus_meta.cc:287] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.506s user 0.011s sys 0.000s
I20251023 02:05:07.594074 23717 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.587263 22028 raft_consensus.cc:2368] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7.
I20251023 02:05:07.545486 23999 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.627794 21956 raft_consensus.cc:1275] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:07.638795 22107 raft_consensus.cc:1275] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:07.663719 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:07.664335 21953 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:07.639817 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.641340 22037 raft_consensus.cc:1240] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:07.644393 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.666678 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.646046 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:07.561164 23816 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.535s user 0.011s sys 0.049s
I20251023 02:05:07.644413 22033 raft_consensus.cc:2393] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:07.620695 23965 raft_consensus.cc:697] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:07.611253 23911 raft_consensus.cc:740] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:07.670282 22035 raft_consensus.cc:2368] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8.
I20251023 02:05:07.676616 22000 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:07.678184 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:07.679193 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:07.680047 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.680897 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.624284 23944 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.684304 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:07.686379 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.686946 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:07.664700 24006 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:07.689795 22028 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:07.627517 23547 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.713915 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.737767 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.668875 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:07.743121 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:07.637138 23739 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.653081 23757 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.751801 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:07.752228 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 128367us
I20251023 02:05:07.640966 23993 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.753247 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.750252 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.757596 22030 raft_consensus.cc:2368] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5.
I20251023 02:05:07.757411 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:07.760654 21953 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:07.764015 22035 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:07.764634 22035 raft_consensus.cc:606] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:07.675243 23814 consensus_queue.cc:1048] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
W20251023 02:05:07.765190 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 88651.8us
I20251023 02:05:07.723178 23755 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:07.774755 22028 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:07.650741 23964 consensus_queue.cc:260] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.765776 22035 raft_consensus.cc:614] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:07.766708 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:07.766155 22032 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:07.709882 23520 consensus_queue.cc:1048] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20251023 02:05:07.747116 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
W20251023 02:05:07.768983 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:07.769079 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:07.771939 22031 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:07.786899 22621 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
W20251023 02:05:07.765775 24017 log.cc:927] Time spent T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.152s user 0.001s sys 0.001s
I20251023 02:05:07.777504 23742 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.797250 23739 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.777397 23755 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.798385 23547 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.799332 23407 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.781527 22032 raft_consensus.cc:1275] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:07.800375 23936 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:07.800778 22032 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:07.778780 22617 raft_consensus.cc:2393] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:07.776324 23828 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.687750 23941 consensus_queue.cc:237] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.804584 23944 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:07.787945 22075 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:07.801644 23757 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:07.779124 23816 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251023 02:05:07.789664 22028 raft_consensus.cc:1275] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20251023 02:05:07.790786 23999 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.761313 23911 consensus_queue.cc:260] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.812332 23911 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:07.799403 23755 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:07.780131 22033 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:07.802619 22032 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:07.816634 22032 raft_consensus.cc:606] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:07.817055 22032 raft_consensus.cc:614] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251023 02:05:07.804081 23994 log.cc:927] Time spent T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.485s user 0.002s sys 0.000s
W20251023 02:05:07.817891 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 487902us
I20251023 02:05:07.789278 22035 raft_consensus.cc:1240] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:07.808225 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:07.813155 22028 pending_rounds.cc:85] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting all ops after (but not including) 2
I20251023 02:05:07.827142 22028 pending_rounds.cc:107] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20251023 02:05:07.827477 22028 raft_consensus.cc:2889] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:07.792562 23993 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:07.813194 23964 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:07.813369 23944 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:05:07.785391 23737 consensus_queue.cc:1059] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20251023 02:05:07.770603 23965 consensus_queue.cc:237] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:07.886979 24014 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20251023 02:05:07.888916 22587 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:07.889321 22030 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:07.891027 22030 raft_consensus.cc:606] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:07.891438 22030 raft_consensus.cc:614] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:07.893867 23717 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:07.883109 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:07.903155 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:07.903331 22033 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:07.896648 23919 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20251023 02:05:07.897315 23774 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.028s sys 0.062s
I20251023 02:05:07.892798 23944 raft_consensus.cc:697] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:07.913996 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1309) took 1435 ms. Trace:
I20251023 02:05:07.914324 22034 rpcz_store.cc:276] 1023 02:05:06.478115 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:06.478284 (+ 169us) service_pool.cc:225] Handling call
1023 02:05:06.536297 (+ 58013us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:06.607552 (+ 71255us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:06.607596 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:06.608161 (+ 565us) log.cc:844] Serialized 26 byte log entry
1023 02:05:06.844456 (+236295us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b48001701d8. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c649a7971 00007f9c6499ea37 00007f9c6499e8ca 00007f9c64a95a0f 00007f9c64a956d7 00007f9c64aa9cfc 00007f9c64aa9ca8 00007f9c64aa9c28 00007f9c64aa9be3 00007f9c64aa8e8c 00007f9c639e5326
1023 02:05:06.917236 (+ 72780us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:06.918336 (+ 1100us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:06.918660 (+ 324us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:07.803505 (+884845us) spinlock_profiling.cc:243] Waited 6.18 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:07.913797 (+110292us) raft_consensus.cc:1685] finished
1023 02:05:07.913817 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:07.913941 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54737,"raft.queue_time_us":71618,"raft.run_cpu_time_us":2636,"raft.run_wall_time_us":230249,"spinlock_wait_cycles":34677248,"thread_start_us":253650,"threads_started":1,"wal-append.queue_time_us":524030}
W20251023 02:05:07.911710 23774 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.838s user 0.028s sys 0.062s
W20251023 02:05:07.928520 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 132903us
I20251023 02:05:07.933727 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:07.934639 22617 raft_consensus.cc:2368] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7.
W20251023 02:05:07.936321 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:07.941381 23944 consensus_queue.cc:237] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.942353 21925 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:07.945021 23845 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:07.945395 23845 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:07.945757 23845 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:07.927521 24015 log.cc:927] Time spent T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
W20251023 02:05:07.930255 24026 log.cc:927] Time spent T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20251023 02:05:07.957754 23986 consensus_queue.cc:1048] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:05:07.958431 23986 consensus_queue.cc:1243] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer ca1362b9d9b34073bf35b6f6c9d1159a log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20251023 02:05:07.959892 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 325073us
I20251023 02:05:07.962340 21928 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:07.963599 22618 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:05:07.964259 23928 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:07.964929 22034 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:07.964799 22075 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:07.965615 22034 raft_consensus.cc:606] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:07.965735 23703 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:07.966423 23964 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:05:07.967051 23964 raft_consensus.cc:697] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:07.970008 23473 raft_consensus.cc:2749] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:07.971968 24024 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:07.972457 24024 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.974437 24024 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:07.968078 23964 consensus_queue.cc:237] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.975622 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:07.976816 22000 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:07.980898 22002 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:07.966244 23703 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:07.964757 23928 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:07.989537 22000 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:07.990943 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:07.999933 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251023 02:05:08.001572 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:05:08.002347 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:05:08.014803 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:08.020665 22107 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251023 02:05:08.024537 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:08.024997 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:08.025342 22108 raft_consensus.cc:2393] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 6: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:07.984290 23774 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:07.984215 24033 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:07.990564 23993 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:08.035866 22077 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.029920 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:08.030701 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:08.038017 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:08.028705 22104 pending_rounds.cc:85] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 1
I20251023 02:05:08.038817 22104 pending_rounds.cc:107] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251023 02:05:08.039182 22104 raft_consensus.cc:2889] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:08.034905 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.034919 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:08.040130 21953 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:08.042595 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.044131 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.044669 22103 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:08.046728 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:08.047492 21955 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:08.049304 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 157596us
I20251023 02:05:07.990721 23737 consensus_queue.cc:1048] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20251023 02:05:08.054121 22077 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:07.990590 23779 consensus_queue.cc:1059] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20251023 02:05:08.056860 23969 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:08.065692 23969 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.057286 23944 raft_consensus.cc:2749] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:08.040598 23742 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:08.067332 23755 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:08.067620 23742 raft_consensus.cc:697] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:08.067916 23969 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:08.068519 23742 consensus_queue.cc:237] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:08.069538 22077 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.069664 23845 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:08.063582 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:08.025840 23816 consensus_queue.cc:1059] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20251023 02:05:08.086529 22077 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.090487 22077 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:08.097333 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 135060us
I20251023 02:05:08.097510 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.098299 22111 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:08.099530 21925 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:08.034976 23828 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20251023 02:05:08.099779 21928 leader_election.cc:400] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5.
I20251023 02:05:08.055239 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:08.101094 21928 leader_election.cc:403] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251023 02:05:08.099568 21955 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:08.102933 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:08.103202 22002 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:08.097114 23755 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:05:08.107118 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 56957.8us
I20251023 02:05:08.105901 21955 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:08.107136 22077 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.108042 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.071514 23963 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:08.036990 23774 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:08.082605 23941 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:08.110147 22002 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:08.113004 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.111888 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:08.095335 24024 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:08.114152 21953 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:08.094532 23717 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:08.115751 22077 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.117401 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:08.117995 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:08.118521 21956 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:08.118566 22077 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.122782 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:08.128480 22077 leader_election.cc:400] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8.
I20251023 02:05:08.128883 22077 leader_election.cc:403] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:05:08.130162 22077 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.110730 23944 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:08.137416 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:08.110828 23923 raft_consensus.cc:2749] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5.
I20251023 02:05:08.188114 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:08.190090 21953 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:08.110719 24043 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:08.191094 22111 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 7
I20251023 02:05:08.140053 22077 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.127126 23520 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:08.137639 22618 raft_consensus.cc:2393] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 3: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
W20251023 02:05:08.196810 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 56853.1us
I20251023 02:05:08.203311 21956 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:08.204655 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.206563 22077 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.129563 23755 raft_consensus.cc:2749] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:08.138062 23965 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 8
I20251023 02:05:08.188007 23732 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:08.233431 21956 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:08.134644 23713 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:08.115356 23963 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:08.252811 21953 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:08.194167 24043 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:08.188266 23799 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:08.257279 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1091) took 1265 ms. Trace:
W20251023 02:05:08.235378 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 112852us
I20251023 02:05:08.270270 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.267804 21962 rpcz_store.cc:276] 1023 02:05:06.992052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:06.992606 (+ 554us) service_pool.cc:225] Handling call
1023 02:05:06.993013 (+ 407us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:06.993412 (+ 399us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:05:07.023240 (+ 29828us) log.cc:844] Serialized 14 byte log entry
1023 02:05:07.965536 (+942296us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:07.978051 (+ 12515us) log.cc:844] Serialized 26 byte log entry
1023 02:05:07.978204 (+ 153us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:07.978406 (+ 202us) log.cc:844] Serialized 14 byte log entry
1023 02:05:07.978635 (+ 229us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:07.978730 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:08.257030 (+278300us) raft_consensus.cc:1685] finished
1023 02:05:08.257070 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:08.257214 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":324754,"prepare.queue_time_us":30389,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":7417,"raft.queue_time_us":337383,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":23718,"thread_start_us":582843,"threads_started":1,"wal-append.queue_time_us":547504}
I20251023 02:05:08.129139 24024 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.204357 23944 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:08.205374 23717 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:08.374517 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:08.413687 22107 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:08.334686 21953 raft_consensus.cc:1275] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251023 02:05:08.242036 23936 raft_consensus.cc:2749] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:05:08.374809 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:08.391499 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 281118us
I20251023 02:05:08.264245 21926 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:08.375454 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.330741 22076 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:08.408900 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.411217 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:08.417157 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 86555.7us
I20251023 02:05:08.418655 22076 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:08.419606 22106 raft_consensus.cc:2368] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:05:08.263036 24036 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:08.270967 23703 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:08.425475 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 188639us
I20251023 02:05:08.426262 22077 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.426599 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.427347 22106 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:08.428854 21925 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:08.253793 24011 consensus_queue.cc:1059] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.162s
I20251023 02:05:08.263439 23993 raft_consensus.cc:697] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:08.267527 23845 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:08.438838 22107 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:08.267127 23721 consensus_queue.cc:1048] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.154s
I20251023 02:05:08.415750 21928 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:08.433140 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1099) took 1095 ms. Trace:
I20251023 02:05:08.442324 21958 rpcz_store.cc:276] 1023 02:05:07.337895 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:07.338017 (+ 122us) service_pool.cc:225] Handling call
1023 02:05:08.432942 (+1094925us) spinlock_profiling.cc:243] Waited 54.6 ms on lock 0x7b54000c4254. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:08.433077 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":125633792}
I20251023 02:05:08.266537 23941 consensus_queue.cc:1059] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.152s
I20251023 02:05:08.433673 22077 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.434788 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.265271 24043 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:08.267618 23928 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:08.420231 23911 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:08.445873 22104 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:08.440735 21925 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:08.422039 22615 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251023 02:05:08.425554 24036 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:08.425006 23732 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:08.424527 23713 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:08.265656 23799 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:08.457468 23991 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.138s user 0.003s sys 0.011s
I20251023 02:05:08.459841 21958 raft_consensus.cc:1275] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:08.459578 23928 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:08.419417 23944 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:08.467559 24045 log.cc:927] Time spent T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20251023 02:05:08.467747 24051 raft_consensus.cc:2749] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:08.439179 23717 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:08.468611 23993 consensus_queue.cc:237] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.458045 23845 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:08.457931 23520 raft_consensus.cc:697] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:08.487149 23991 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.169s user 0.003s sys 0.011s
I20251023 02:05:08.459749 23911 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:08.498600 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 65022.2us
I20251023 02:05:08.500365 24063 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:08.504103 22107 raft_consensus.cc:1275] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251023 02:05:08.488562 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 378504us
I20251023 02:05:08.476668 23703 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:08.467572 23963 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:08.467576 24036 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:08.490653 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 226542us
I20251023 02:05:08.478457 23774 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:08.483989 23928 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:08.484097 23721 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:08.527215 23888 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:08.522562 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 46313.5us
W20251023 02:05:08.528661 24027 log.cc:927] Time spent T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.699s user 0.001s sys 0.003s
I20251023 02:05:08.538591 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1083) took 1412 ms. Trace:
I20251023 02:05:08.539230 24033 raft_consensus.cc:3055] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:08.628089 22037 rpcz_store.cc:276] 1023 02:05:07.126054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:07.126364 (+ 310us) service_pool.cc:225] Handling call
1023 02:05:07.635123 (+508759us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:07.669921 (+ 34798us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:08.538416 (+868495us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b54002a8fd0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:08.538537 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1169286528}
W20251023 02:05:08.543133 24025 log.cc:927] Time spent T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.724s user 0.003s sys 0.000s
I20251023 02:05:08.544659 23779 raft_consensus.cc:3055] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:08.564530 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:08.636489 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:08.616961 23928 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:08.610805 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.508208 23732 raft_consensus.cc:697] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:08.510413 23713 raft_consensus.cc:697] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:08.638055 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:08.639256 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:08.640528 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:08.640903 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:08.608206 24048 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.009s sys 0.007s
I20251023 02:05:08.651149 22077 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.658547 22002 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:08.661394 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:08.661602 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:08.665505 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:08.662137 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:08.666786 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:08.660682 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:08.671203 22032 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:08.672240 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:08.673221 22032 raft_consensus.cc:2410] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:05:08.673949 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:08.674454 22029 raft_consensus.cc:1275] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:08.675058 24057 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:08.675518 24057 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:08.629789 24033 raft_consensus.cc:740] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:08.623302 23721 consensus_queue.cc:1048] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20251023 02:05:08.657027 23407 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:08.563450 24063 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.633242 23779 raft_consensus.cc:740] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:08.662957 24048 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.578s user 0.010s sys 0.007s
I20251023 02:05:08.681247 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:08.727973 22109 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:08.696822 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:08.659845 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:08.614459 24053 consensus_queue.cc:1048] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20251023 02:05:08.702795 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:08.702301 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 43821.8us
I20251023 02:05:08.661362 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:08.699486 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:08.723421 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:08.629603 23888 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:08.668365 24024 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:08.759343 22111 consensus_meta.cc:287] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.565s user 0.007s sys 0.002s
I20251023 02:05:08.739856 22110 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:08.711198 23937 raft_consensus.cc:3055] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:08.802819 22111 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:08.804862 22622 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20251023 02:05:08.710904 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:08.717468 23774 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:08.819074 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:08.819449 22031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1113) took 1281 ms (client timeout 1525 ms). Trace:
I20251023 02:05:08.807338 22622 pending_rounds.cc:85] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting all ops after (but not including) 4
I20251023 02:05:08.820073 22622 pending_rounds.cc:107] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20251023 02:05:08.820364 22622 raft_consensus.cc:2889] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:08.717391 23963 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:08.817684 22109 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:08.820608 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:08.819788 22031 rpcz_store.cc:269] 1023 02:05:07.524378 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:07.645759 (+121381us) service_pool.cc:225] Handling call
1023 02:05:08.805377 (+1159618us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54002a80d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:08.805477 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":239437056}
I20251023 02:05:08.820658 22622 pending_rounds.cc:107] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20251023 02:05:08.658104 23520 consensus_queue.cc:237] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.662122 23732 consensus_queue.cc:237] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.779606 23703 raft_consensus.cc:697] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:08.826640 22622 raft_consensus.cc:2889] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:08.809309 23999 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:08.852072 21957 raft_consensus.cc:1275] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:08.806933 23937 raft_consensus.cc:740] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:08.721292 23407 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.853844 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:08.940755 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 289679us
I20251023 02:05:08.859426 24057 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:08.863587 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:08.931425 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 265174us
W20251023 02:05:08.846269 24028 log.cc:927] Time spent T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 1.018s user 0.003s sys 0.000s
I20251023 02:05:08.818646 24024 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:08.844168 24068 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:08.819169 24051 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:08.858111 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:08.823832 24065 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:08.941325 21958 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
W20251023 02:05:08.942054 22077 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.525s (SENT) [suppressed 11 similar messages]
I20251023 02:05:08.837532 23755 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:08.948011 22108 raft_consensus.cc:2410] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:05:08.824090 24055 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:08.948545 22110 raft_consensus.cc:1240] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20251023 02:05:08.844033 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:08.942126 23911 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:08.840034 23774 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:08.862720 24034 log.cc:927] Time spent T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.907s user 0.000s sys 0.014s
I20251023 02:05:08.945459 24077 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:08.957737 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 94535.1us
I20251023 02:05:08.961987 22028 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:08.938961 23986 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:08.940618 23999 raft_consensus.cc:697] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:08.844765 23963 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.939234 22109 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251023 02:05:08.825722 23713 consensus_queue.cc:237] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.986663 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:08.953338 21926 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:08.955354 24049 log.cc:927] Time spent T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.442s user 0.000s sys 0.001s
W20251023 02:05:08.992000 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 256625us
I20251023 02:05:08.941951 23941 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:08.958549 23814 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:08.955813 24042 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20251023 02:05:08.958370 24024 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.999512 22590 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:08.853129 23779 consensus_queue.cc:260] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.008556 22076 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:09.008184 22109 raft_consensus.cc:1275] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:08.978268 23944 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:08.973488 24065 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.006081 24057 consensus_queue.cc:1059] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20251023 02:05:09.006059 23703 consensus_queue.cc:237] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.975348 22623 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:08.853191 24033 consensus_queue.cc:260] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.983227 24071 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:09.028589 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:08.977165 22102 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:09.023540 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:09.007208 24058 log.cc:927] Time spent T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.373s user 0.001s sys 0.000s
I20251023 02:05:09.024456 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.032991 22035 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:09.030994 21953 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:09.007817 24054 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:08.963106 24031 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20251023 02:05:09.028338 22616 raft_consensus.cc:2393] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:09.009404 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:09.016476 24065 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:09.038978 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:09.040304 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 98276.1us
I20251023 02:05:08.973672 23755 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.067278 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1098) took 1729 ms. Trace:
W20251023 02:05:09.015393 24070 log.cc:927] Time spent T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20251023 02:05:09.021435 24037 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:09.014192 24062 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:09.015744 24024 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.017882 23888 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.021553 24064 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:08.966383 24051 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.984665 23986 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:08.982594 24077 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:05:09.021395 24050 consensus_queue.cc:1048] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
I20251023 02:05:09.056829 24063 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:09.099757 24043 consensus_meta.cc:287] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.638s user 0.009s sys 0.000s
I20251023 02:05:09.058383 23963 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.060055 24053 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:09.170501 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1109) took 1185 ms. Trace:
I20251023 02:05:09.170955 21956 rpcz_store.cc:276] 1023 02:05:07.985384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:08.255121 (+269737us) service_pool.cc:225] Handling call
1023 02:05:08.393487 (+138366us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:08.412479 (+ 18992us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:08.412516 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:08.413633 (+ 1117us) log.cc:844] Serialized 26 byte log entry
1023 02:05:08.470285 (+ 56652us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:08.470334 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:08.470396 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:08.551427 (+ 81031us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b48000b22d8. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c649a7971 00007f9c6499ea37 00007f9c6499e8ca 00007f9c64a95a0f 00007f9c64a956d7 00007f9c64aa9cfc 00007f9c64aa9ca8 00007f9c64aa9c28 00007f9c64aa9be3 00007f9c64aa8e8c 00007f9c639e5326
1023 02:05:09.146412 (+594985us) raft_consensus.cc:1685] finished
1023 02:05:09.146433 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:09.170435 (+ 24002us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":111710,"raft.queue_time_us":27242,"raft.run_cpu_time_us":2983,"raft.run_wall_time_us":118934,"spinlock_wait_cycles":10652032,"thread_start_us":25319,"threads_started":1,"wal-append.queue_time_us":93081}
I20251023 02:05:09.062673 22618 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251023 02:05:09.062980 23936 raft_consensus.cc:3055] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
W20251023 02:05:09.064349 24044 log.cc:927] Time spent T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.579s user 0.001s sys 0.000s
W20251023 02:05:09.022781 22617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 218) took 1195 ms (client timeout 1530 ms). Trace:
I20251023 02:05:08.998842 24067 consensus_queue.cc:1048] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.063s
I20251023 02:05:09.042019 22590 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:09.204291 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 162392us
I20251023 02:05:09.067786 21959 rpcz_store.cc:276] 1023 02:05:07.337601 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:07.341806 (+ 4205us) service_pool.cc:225] Handling call
1023 02:05:07.342102 (+ 296us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:07.361293 (+ 19191us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:07.361322 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:07.617965 (+256643us) log.cc:844] Serialized 52 byte log entry
1023 02:05:07.890329 (+272364us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:07.890372 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:07.890435 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:08.567604 (+677169us) spinlock_profiling.cc:243] Waited 8.63 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:09.048275 (+480671us) raft_consensus.cc:1685] finished
1023 02:05:09.048295 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:09.067196 (+ 18901us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5673,"raft.queue_time_us":19786,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":30981,"spinlock_wait_cycles":19855104,"thread_start_us":216691,"threads_started":1,"wal-append.queue_time_us":259675}
I20251023 02:05:08.976822 23937 consensus_queue.cc:260] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.048378 24061 raft_consensus.cc:3055] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:09.097744 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.145819 24050 consensus_queue.cc:1243] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251023 02:05:09.010003 23814 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:08.977224 24055 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.105142 22111 raft_consensus.cc:1275] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251023 02:05:09.016862 23999 consensus_queue.cc:237] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.149209 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 139) took 1208 ms. Trace:
I20251023 02:05:09.026795 23941 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.271728 21928 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:09.182375 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.272892 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.273082 21928 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.201042 21953 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:09.037681 24071 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.046121 23779 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:09.260564 22109 raft_consensus.cc:2393] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:09.264940 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.278093 22108 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:09.266218 22111 pending_rounds.cc:85] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 1
I20251023 02:05:09.278867 22111 pending_rounds.cc:107] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251023 02:05:09.279189 22111 raft_consensus.cc:2889] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:09.157560 23703 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:09.270363 21961 rpcz_store.cc:276] 1023 02:05:07.941241 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:07.947890 (+ 6649us) service_pool.cc:225] Handling call
1023 02:05:08.003608 (+ 55718us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:08.019038 (+ 15430us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:08.019086 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:08.026246 (+ 7160us) log.cc:844] Serialized 26 byte log entry
1023 02:05:08.204928 (+178682us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:08.413235 (+208307us) log.cc:844] Serialized 14 byte log entry
1023 02:05:08.470513 (+ 57278us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:08.470585 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:09.149008 (+678423us) raft_consensus.cc:1685] finished
1023 02:05:09.149028 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:09.149154 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15026,"prepare.queue_time_us":204402,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":4386,"raft.queue_time_us":51387,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":6283,"thread_start_us":186049,"threads_started":2,"wal-append.queue_time_us":406176}
W20251023 02:05:09.192126 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 183578us
I20251023 02:05:09.275041 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.170830 24053 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:09.283150 22107 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:09.268556 23407 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.200964 24063 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:09.176205 24024 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:09.176003 23936 raft_consensus.cc:740] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:09.259915 24061 raft_consensus.cc:740] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:09.284264 24086 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:09.282444 23473 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:09.286605 24053 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:09.146800 24064 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.157464 24043 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.290285 23774 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:09.291091 23937 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:09.308709 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.325537 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.107146 24062 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:09.317057 24075 log.cc:927] Time spent T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.311s user 0.001s sys 0.000s
W20251023 02:05:09.317823 24020 log.cc:927] Time spent T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 1.489s user 0.007s sys 0.004s
I20251023 02:05:09.318145 21956 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251023 02:05:09.319108 24071 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:09.319821 23964 raft_consensus.cc:3055] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
W20251023 02:05:09.319940 24083 log.cc:927] Time spent T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20251023 02:05:09.320681 24055 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:09.320914 24082 log.cc:927] Time spent T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
I20251023 02:05:09.321885 22623 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251023 02:05:09.342621 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:09.204365 22617 rpcz_store.cc:269] 1023 02:05:07.813976 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:07.814446 (+ 470us) service_pool.cc:225] Handling call
1023 02:05:08.986863 (+1172417us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b54005ce9d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:09.006216 (+ 19353us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33850240}
I20251023 02:05:09.295667 24100 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:09.368847 24100 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:09.369156 24029 log.cc:927] Time spent T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 1.550s user 0.002s sys 0.011s
I20251023 02:05:09.369248 24100 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:09.355836 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.307103 23473 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:09.289134 23703 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.385800 21928 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.319108 23799 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.367108 22108 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:09.370426 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1106) took 1970 ms. Trace:
I20251023 02:05:09.371412 24084 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20251023 02:05:09.372622 24096 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20251023 02:05:09.401955 23941 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:09.373296 24088 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20251023 02:05:09.373487 24051 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.373909 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:09.406500 21956 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:09.354149 23964 raft_consensus.cc:740] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
W20251023 02:05:09.374536 24079 log.cc:927] Time spent T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.333s user 0.001s sys 0.000s
W20251023 02:05:09.373219 23928 consensus_meta.cc:287] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.707s user 0.008s sys 0.002s
I20251023 02:05:09.296463 24063 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:09.298804 24024 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:09.351440 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 68745us
I20251023 02:05:09.423846 22108 raft_consensus.cc:1240] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:09.426489 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:09.427428 22623 raft_consensus.cc:2410] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251023 02:05:09.427819 24102 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:09.428318 24102 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.428884 24064 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.429397 24064 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:09.420732 23928 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.429893 24064 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.431602 23928 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:09.431807 24064 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.368254 22000 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:09.432679 23965 consensus_meta.cc:287] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.176s user 0.012s sys 0.000s
I20251023 02:05:09.433074 23965 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8.
I20251023 02:05:09.377496 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.433672 22103 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:09.376675 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1429) took 1393 ms. Trace:
I20251023 02:05:09.352669 23780 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:09.395429 23703 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:09.436901 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:09.388789 23473 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:09.398545 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.375515 23755 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:09.398553 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.392052 21954 rpcz_store.cc:276] 1023 02:05:07.399846 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:07.400121 (+ 275us) service_pool.cc:225] Handling call
1023 02:05:07.411468 (+ 11347us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:07.608217 (+196749us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:07.608265 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:07.608799 (+ 534us) log.cc:844] Serialized 26 byte log entry
1023 02:05:07.756726 (+147927us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:07.756766 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:07.756843 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:09.324575 (+1567732us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:09.370133 (+ 45558us) raft_consensus.cc:1685] finished
1023 02:05:09.370157 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:09.370368 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71369984,"thread_start_us":147562,"threads_started":1,"wal-append.queue_time_us":203545}
I20251023 02:05:09.400004 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.404593 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.438987 22028 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:09.412310 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1323) took 2291 ms. Trace:
I20251023 02:05:09.439870 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.392385 24092 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:09.412921 22077 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:09.413734 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.364063 24097 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:09.434247 21955 rpcz_store.cc:276] 1023 02:05:07.983764 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:08.130892 (+147128us) service_pool.cc:225] Handling call
1023 02:05:08.131244 (+ 352us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:08.489242 (+357998us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:08.489284 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:08.510293 (+ 21009us) log.cc:844] Serialized 26 byte log entry
1023 02:05:08.691164 (+180871us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:08.691220 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:08.691314 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:09.376425 (+685111us) raft_consensus.cc:1685] finished
1023 02:05:09.376452 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:09.376606 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":246266,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":9724,"thread_start_us":432851,"threads_started":2,"wal-append.queue_time_us":386389}
I20251023 02:05:09.438341 22030 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:09.449024 22110 raft_consensus.cc:1275] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251023 02:05:09.450042 22108 raft_consensus.cc:1240] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:09.439572 22105 rpcz_store.cc:276] 1023 02:05:07.111484 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:07.111653 (+ 169us) service_pool.cc:225] Handling call
1023 02:05:07.111929 (+ 276us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:07.112241 (+ 312us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:07.112269 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:07.254430 (+142161us) spinlock_profiling.cc:243] Waited 92.2 ms on lock 0x7b2c0005f718. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5e3949b1 00007f9c5e3a3fe2 00007f9c5fad4701 00007f9c5fa98fda 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22
1023 02:05:07.313633 (+ 59203us) log.cc:844] Serialized 26 byte log entry
1023 02:05:07.723776 (+410143us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:07.723828 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:07.723932 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:09.264989 (+1541057us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:09.402915 (+137926us) raft_consensus.cc:1685] finished
1023 02:05:09.402941 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:09.403085 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":330,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":3284,"spinlock_wait_cycles":270457984,"thread_start_us":409262,"threads_started":1,"wal-append.queue_time_us":496668}
I20251023 02:05:09.451102 21955 raft_consensus.cc:1240] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
W20251023 02:05:09.412407 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 58757.6us
I20251023 02:05:09.441761 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.445888 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.445677 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:09.418835 23986 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.419171 24077 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.462419 22036 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:09.463048 21954 raft_consensus.cc:1275] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:09.463310 22029 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20251023 02:05:09.464393 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:09.454253 21955 raft_consensus.cc:1275] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251023 02:05:09.442417 23780 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.457114 24043 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.457346 24063 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:09.458768 24062 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.482661 24043 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:09.483232 24043 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.484822 24024 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.453413 23814 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:09.485945 24043 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:09.454972 24097 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.488756 24097 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:09.482271 23473 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.464599 21954 pending_rounds.cc:85] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 2
W20251023 02:05:09.448632 23845 consensus_meta.cc:287] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.947s user 0.008s sys 0.003s
I20251023 02:05:09.442775 24092 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:09.490810 23845 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:09.490996 24092 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:09.491762 21957 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20251023 02:05:09.492599 24090 log.cc:927] Time spent T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.436s user 0.000s sys 0.001s
W20251023 02:05:09.474928 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 62093.2us
I20251023 02:05:09.475544 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 141) took 1533 ms. Trace:
I20251023 02:05:09.472293 24105 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:09.360491 24061 consensus_queue.cc:260] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.477744 23721 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:09.477968 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:09.494400 24105 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.495234 22623 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:09.494062 21960 rpcz_store.cc:276] 1023 02:05:07.941827 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:07.974942 (+ 33115us) service_pool.cc:225] Handling call
1023 02:05:07.975262 (+ 320us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:08.027160 (+ 51898us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:08.027206 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:08.027266 (+ 60us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:08.027309 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:09.475336 (+1448027us) spinlock_profiling.cc:243] Waited 51.7 ms on lock 0x7b54000be5d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:09.475391 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:09.475496 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":118828928}
W20251023 02:05:09.497108 22075 leader_election.cc:336] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255): Timed out: RequestConsensusVote RPC to 127.20.81.4:33255 timed out after 1.530s (SENT)
I20251023 02:05:09.467708 23964 consensus_queue.cc:260] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.423213 24074 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.044s
I20251023 02:05:09.497701 21954 pending_rounds.cc:107] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251023 02:05:09.500227 21925 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.468768 22028 raft_consensus.cc:1275] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:09.500502 21926 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
I20251023 02:05:09.501343 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:09.504876 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 136740us
W20251023 02:05:09.480922 24036 consensus_meta.cc:287] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.857s user 0.011s sys 0.000s
I20251023 02:05:09.512253 21928 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:09.517334 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:09.518332 21925 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.522706 21928 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:09.493757 22077 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:09.519101 21954 raft_consensus.cc:2889] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:09.523864 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.524909 22077 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:09.521384 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.519132 22000 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.519207 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.527324 22105 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:09.519686 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.528184 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:09.528795 22105 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:09.529114 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.530967 22077 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:09.532218 22077 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:09.533010 22077 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:09.534639 21925 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:09.536090 21925 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.538655 22033 raft_consensus.cc:2393] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:09.540494 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:09.541254 22077 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:09.542392 21962 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
W20251023 02:05:09.575644 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:09.584364 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.542876 22077 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:09.564851 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 45821.2us
W20251023 02:05:09.559005 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:09.586073 22000 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.570976 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.587399 22000 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.587520 22035 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:09.589011 22105 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:09.590243 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:09.593376 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.516426 22102 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:09.584136 24042 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 02:05:09.523730 24063 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:09.617728 22108 raft_consensus.cc:2410] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:05:09.617332 21925 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:09.619351 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.630831 22109 raft_consensus.cc:2410] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 7 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:05:09.545087 24062 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:09.546020 23964 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:09.517107 24110 raft_consensus.cc:2749] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:09.538061 24071 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:09.527187 23845 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:09.563463 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:09.539366 24036 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.658454 22034 raft_consensus.cc:2393] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:09.657984 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:09.659125 24036 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.563319 24077 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:09.664369 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.538085 23721 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251023 02:05:09.630872 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 88059.6us
I20251023 02:05:09.538255 23944 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:09.634660 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.590497 23814 raft_consensus.cc:2749] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:09.592662 23703 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:09.642750 21925 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.679100 21925 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:09.680163 21925 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.681802 24055 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:09.681576 23986 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:09.683385 23986 raft_consensus.cc:697] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:09.685052 23703 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:09.684631 23986 consensus_queue.cc:237] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.681547 21925 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.544960 24114 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:09.617270 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:09.689411 22623 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:09.645381 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.691380 22109 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:09.563803 23717 consensus_queue.cc:1048] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20251023 02:05:09.593497 24107 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:09.651435 23845 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:09.538159 24097 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:09.653271 24105 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:09.539052 24043 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:09.642695 23799 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:09.647653 24063 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:09.647876 24084 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:09.668728 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:09.668813 24060 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:09.665890 24077 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:09.700338 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 72277us
I20251023 02:05:09.699486 23845 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:09.706231 21925 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.706418 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:09.670106 24102 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:09.695138 24114 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:09.722744 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:09.708025 24051 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:09.719610 21925 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:09.701833 24063 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:09.702096 24043 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
W20251023 02:05:09.731554 24113 log.cc:927] Time spent T 00000000000000000000000000000000 P aec15eab11a445ed83475b43469d1669: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
I20251023 02:05:09.727706 21925 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:09.711760 24077 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:09.707847 23799 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.749285 24062 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:09.722362 23473 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:09.755131 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:09.732062 24063 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:09.759174 24107 consensus_queue.cc:1059] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:05:09.732741 24043 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:09.732271 24051 raft_consensus.cc:2804] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:09.723515 23703 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:09.767113 23799 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.780462 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:09.801474 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:09.781957 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:09.813573 22109 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
W20251023 02:05:09.812520 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:09.814800 21955 raft_consensus.cc:2393] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
W20251023 02:05:09.814936 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:09.815325 21925 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:09.816171 24062 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:09.802249 21957 raft_consensus.cc:1275] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:09.816560 24062 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:09.816970 24062 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:09.817356 24109 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:09.817905 24109 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.790663 22028 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:09.821481 22104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1455) took 1211 ms. Trace:
I20251023 02:05:09.827754 22104 rpcz_store.cc:276] 1023 02:05:08.610281 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:08.610479 (+ 198us) service_pool.cc:225] Handling call
1023 02:05:09.813414 (+1202935us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b54001dc254. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:09.821426 (+ 8012us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1757453184}
I20251023 02:05:09.825450 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:09.827131 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:09.819906 24020 log.cc:927] Time spent T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20251023 02:05:09.801975 23814 raft_consensus.cc:3055] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:09.774029 24102 consensus_queue.cc:1048] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.067s
I20251023 02:05:09.739298 24114 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.774523 24051 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:09.822945 23845 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.889911 21928 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:09.834568 24055 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:09.874753 22616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34806 (request call id 307) took 1211 ms. Trace:
I20251023 02:05:09.893901 24042 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election won for term 6
W20251023 02:05:09.894323 24104 log.cc:927] Time spent T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.200s user 0.001s sys 0.001s
I20251023 02:05:09.789029 24111 consensus_queue.cc:1048] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.084s
I20251023 02:05:09.902141 22616 rpcz_store.cc:276] 1023 02:05:08.663848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:08.666670 (+ 2822us) service_pool.cc:225] Handling call
1023 02:05:09.859315 (+1192645us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b540067cfd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:09.874683 (+ 15368us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":469163904}
W20251023 02:05:09.875641 24053 consensus_meta.cc:287] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.548s user 0.009s sys 0.000s
W20251023 02:05:09.852133 23779 consensus_meta.cc:287] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.565s user 0.010s sys 0.003s
I20251023 02:05:09.872715 23963 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:09.850303 23814 raft_consensus.cc:740] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:09.771253 24092 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.922288 24092 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.915076 24042 raft_consensus.cc:697] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:09.923348 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:09.923816 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:09.923982 24042 consensus_queue.cc:237] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:09.925464 23814 consensus_queue.cc:260] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.925936 21925 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:09.926426 23937 consensus_meta.cc:287] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.577s user 0.007s sys 0.003s
I20251023 02:05:09.926676 23814 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:09.926957 22032 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:09.927117 23703 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:09.891044 24051 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:09.929208 24114 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:09.837512 23473 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:09.932250 23845 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:09.937997 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:09.944340 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:09.946933 22616 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:09.850150 23799 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:09.950467 22617 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.002s sys 0.021s
I20251023 02:05:09.910857 24055 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.944988 23963 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:09.956611 24100 consensus_meta.cc:287] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.557s user 0.009s sys 0.000s
W20251023 02:05:09.960144 22617 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.512s user 0.002s sys 0.021s
W20251023 02:05:09.966117 23297 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.186s user 0.002s sys 0.028s
I20251023 02:05:09.957317 23799 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.963912 24109 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:09.917758 24053 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:09.949713 23473 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:09.970073 23999 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.038s sys 0.102s
W20251023 02:05:09.989945 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:09.970573 24060 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
W20251023 02:05:10.000542 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:09.987520 23297 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.255s user 0.002s sys 0.028s
W20251023 02:05:10.004554 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 417231us
W20251023 02:05:10.001169 23999 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s user 0.038s sys 0.102s
I20251023 02:05:10.010447 22000 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:10.011199 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:09.986202 22623 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:10.012090 23703 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:09.980489 24100 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:10.031363 22000 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:10.033083 22000 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:10.034891 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1456) took 1403 ms. Trace:
I20251023 02:05:10.006985 24109 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:10.035506 22106 rpcz_store.cc:276] 1023 02:05:08.631580 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:08.632044 (+ 464us) service_pool.cc:225] Handling call
1023 02:05:08.652041 (+ 19997us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:08.652342 (+ 301us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:08.652370 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:08.652820 (+ 450us) log.cc:844] Serialized 26 byte log entry
1023 02:05:08.837289 (+184469us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:09.731858 (+894569us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b18004549c0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9cdd37 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9e8445 00007f9c63488b27 00007f9c5faa3537 00007f9c5fab2c4f 00007f9c5fab2bc8
1023 02:05:10.012144 (+280286us) log.cc:844] Serialized 14 byte log entry
1023 02:05:10.012408 (+ 264us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:10.012496 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:10.012536 (+ 40us) raft_consensus.cc:1685] finished
1023 02:05:10.034702 (+ 22166us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:10.034837 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17670,"prepare.queue_time_us":145059,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":148254,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":25387,"spinlock_wait_cycles":3682688,"thread_start_us":413310,"threads_started":2,"wal-append.queue_time_us":313789}
I20251023 02:05:10.043387 22000 leader_election.cc:400] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:05:10.030685 23717 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:10.044651 22000 leader_election.cc:403] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:05:10.012776 23713 raft_consensus.cc:2749] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:10.049654 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:10.033972 24111 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:10.035032 24122 raft_consensus.cc:2804] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:10.054936 22106 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:10.036113 23999 raft_consensus.cc:2749] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:10.059777 21925 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:10.045563 24080 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:10.068859 22105 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:10.005667 24060 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:10.055002 23963 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.090402 22000 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:10.093248 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:10.096735 22000 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:10.066318 24107 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 5
I20251023 02:05:10.058797 24122 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:10.067785 23986 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:10.045011 23717 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.046757 24109 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:10.036593 23703 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:10.101462 22107 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.012s sys 0.023s
I20251023 02:05:10.087960 24060 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:10.104621 22107 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s user 0.012s sys 0.023s
I20251023 02:05:10.105073 22000 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:10.068075 24084 consensus_queue.cc:1048] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
W20251023 02:05:10.101135 24125 log.cc:927] Time spent T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20251023 02:05:10.112658 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:10.098018 23999 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:10.101891 24122 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:10.102505 23986 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:10.119647 21925 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:10.092164 24080 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.111189 24110 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:10.110488 24102 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:10.111140 24055 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.142202 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:10.129101 23986 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:10.144069 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 285406us
I20251023 02:05:10.144327 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:10.134059 24110 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:10.151408 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:10.134632 24052 raft_consensus.cc:2749] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:10.138423 24053 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:10.163753 24123 log.cc:927] Time spent T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20251023 02:05:10.154667 24110 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:10.215636 24111 consensus_queue.cc:1059] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:05:10.218199 24111 raft_consensus.cc:3055] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:10.218341 24116 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:10.218555 24111 raft_consensus.cc:740] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:10.218843 24116 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:10.219372 24111 consensus_queue.cc:260] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.220700 24111 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:10.163420 24102 consensus_queue.cc:1048] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20251023 02:05:10.220881 24116 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.216998 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1349) took 1115 ms. Trace:
I20251023 02:05:10.223744 22622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 310) took 1018 ms. Trace:
W20251023 02:05:10.216420 24137 log.cc:927] Time spent T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20251023 02:05:10.158361 24133 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20251023 02:05:10.224449 23799 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.160215 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:10.152510 24063 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
W20251023 02:05:10.226013 22034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1121) took 1590 ms (client timeout 1869 ms). Trace:
I20251023 02:05:10.226744 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:10.227034 23717 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:10.227197 24063 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.227520 22106 raft_consensus.cc:2393] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:10.217247 24080 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.224114 22622 rpcz_store.cc:276] 1023 02:05:09.205253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:09.205864 (+ 611us) service_pool.cc:225] Handling call
1023 02:05:09.906045 (+700181us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b5800214598. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:05:09.906292 (+ 247us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:09.906701 (+ 409us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:09.906740 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:09.907256 (+ 516us) log.cc:844] Serialized 26 byte log entry
1023 02:05:10.119959 (+212703us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:10.120013 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:10.120103 (+ 90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:10.223461 (+103358us) raft_consensus.cc:1685] finished
1023 02:05:10.223483 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:10.223688 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1275,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":3258,"spinlock_wait_cycles":4279680,"thread_start_us":212315,"threads_started":1,"wal-append.queue_time_us":243523}
I20251023 02:05:10.229146 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36488 (request call id 147) took 1206 ms. Trace:
I20251023 02:05:10.229498 21956 raft_consensus.cc:1240] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 6. Ops: [4.5-4.5]
I20251023 02:05:10.229506 21953 rpcz_store.cc:276] 1023 02:05:09.023203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:09.038741 (+ 15538us) service_pool.cc:225] Handling call
1023 02:05:10.228983 (+1190242us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400218c54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:10.229091 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":343145216}
I20251023 02:05:10.231760 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:10.232340 22106 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:10.226390 22034 rpcz_store.cc:269] 1023 02:05:08.635768 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:08.637800 (+ 2032us) service_pool.cc:225] Handling call
1023 02:05:10.225842 (+1588042us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54003a9b54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:10.225964 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2281713536}
W20251023 02:05:10.233314 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 1.08367e+06us
I20251023 02:05:10.222833 22107 rpcz_store.cc:276] 1023 02:05:09.101988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:09.293450 (+191462us) service_pool.cc:225] Handling call
1023 02:05:09.293840 (+ 390us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:09.413313 (+119473us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:09.413355 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:09.419261 (+ 5906us) log.cc:844] Serialized 26 byte log entry
1023 02:05:10.105885 (+686624us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:10.105935 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:10.106014 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:10.216766 (+110752us) raft_consensus.cc:1685] finished
1023 02:05:10.216787 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:10.216928 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39625,"raft.queue_time_us":73297,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":13220,"thread_start_us":723123,"threads_started":2,"wal-append.queue_time_us":692325}
I20251023 02:05:10.236081 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:10.154973 23473 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:10.237093 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:10.237268 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:10.237758 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:10.240821 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:10.241310 22000 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:10.239895 21962 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:05:10.241415 21957 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:05:10.252359 23818 consensus_queue.cc:1048] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:10.165915 24100 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:10.260906 23703 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.260785 23818 consensus_queue.cc:1048] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:05:10.267529 22109 raft_consensus.cc:1275] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251023 02:05:10.268464 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:10.270076 22617 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251023 02:05:10.271962 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:10.242879 23717 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20251023 02:05:10.273901 22077 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:10.274986 23717 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:10.275414 23717 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:10.275817 23717 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:10.278527 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:55760 (request call id 5) took 1643 ms. Trace:
I20251023 02:05:10.279047 22037 rpcz_store.cc:276] 1023 02:05:08.635488 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:08.635675 (+ 187us) service_pool.cc:225] Handling call
1023 02:05:10.278231 (+1642556us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54002ea754. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9f2c5 00007f9c64a442e9 00007f9c5dfe0334 00007f9c5dfe02c2 00007f9c5dfe01f1 00007f9c5dfe016c 00007f9c5dfdf422 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:10.278440 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2902654848}
I20251023 02:05:10.279735 23986 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.281745 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55770 (request call id 191) took 1259 ms. Trace:
I20251023 02:05:10.283596 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:10.284291 22030 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:10.240641 24129 log.cc:927] Time spent T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.095s user 0.001s sys 0.008s
I20251023 02:05:10.289342 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:10.288314 22106 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:10.288341 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:10.282203 22032 rpcz_store.cc:276] 1023 02:05:09.022527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:09.024242 (+ 1715us) service_pool.cc:225] Handling call
1023 02:05:10.281528 (+1257286us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x7b540040c9d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:10.281675 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2044074752}
I20251023 02:05:10.288686 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:10.292171 23964 consensus_queue.cc:1059] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:10.292754 21925 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:10.293545 24052 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:10.294443 24051 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:10.295096 24052 raft_consensus.cc:697] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:10.295181 22616 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:10.295720 21961 raft_consensus.cc:2393] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:10.296216 23965 consensus_queue.cc:1048] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:10.300714 22106 raft_consensus.cc:1275] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251023 02:05:10.301549 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:10.302130 22037 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:10.290730 22109 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
W20251023 02:05:10.297185 24043 consensus_meta.cc:287] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.526s user 0.008s sys 0.009s
I20251023 02:05:10.296053 24052 consensus_queue.cc:237] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:05:10.303859 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:10.303859 24043 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.295202 23717 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.305721 24043 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.250309 24080 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:10.306120 22104 raft_consensus.cc:1275] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:10.303220 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:10.307022 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:10.307516 22032 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:10.269393 24107 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5.
I20251023 02:05:10.307785 23717 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:10.309811 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:10.315330 22106 raft_consensus.cc:1240] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:10.317910 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:10.318252 24140 log.cc:927] Time spent T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20251023 02:05:10.319129 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:10.567111 21928 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:10.567444 22034 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:05:10.567994 23703 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:10.568495 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:10.569219 21928 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:10.412626 22030 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:10.570996 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:10.370620 21958 raft_consensus.cc:1275] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20251023 02:05:10.434230 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 194567us
I20251023 02:05:10.573642 22589 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
I20251023 02:05:10.350848 22032 raft_consensus.cc:2410] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20251023 02:05:10.574501 22589 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
I20251023 02:05:10.578277 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:10.503159 22036 raft_consensus.cc:1275] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:10.450150 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:10.583848 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:10.399823 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:10.579375 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:10.598459 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 357039us
I20251023 02:05:10.601037 21926 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:10.331670 24077 consensus_meta.cc:287] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.577s user 0.011s sys 0.006s
I20251023 02:05:10.311528 24109 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.309270 23986 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.601576 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:10.434633 24031 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.378s user 0.014s sys 0.009s
W20251023 02:05:10.472072 24142 log.cc:927] Time spent T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20251023 02:05:10.611367 22028 raft_consensus.cc:1275] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:10.341221 24086 consensus_queue.cc:1048] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.279s
I20251023 02:05:10.627560 24086 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:10.628209 24086 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.630585 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:10.631219 22033 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:10.613147 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:10.329404 24024 raft_consensus.cc:2749] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:10.614554 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:10.642340 22030 raft_consensus.cc:2393] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:10.615731 22002 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:10.413249 24092 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:10.643546 24092 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.615957 23717 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:10.413470 24143 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:10.472956 24043 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:10.356510 23964 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:10.594511 23703 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:10.341280 24055 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:10.473430 23923 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:10.332060 22615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 312) took 1229 ms. Trace:
I20251023 02:05:10.632500 22107 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:10.635215 21908 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 818907d63f1f4d84ba267ebe6ee837e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
W20251023 02:05:10.619030 24031 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.563s user 0.014s sys 0.019s
I20251023 02:05:10.602523 23739 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:10.547514 24110 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.441109 24107 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:10.639065 24062 consensus_meta.cc:287] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.822s user 0.010s sys 0.010s
I20251023 02:05:10.602703 23755 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20251023 02:05:10.412588 23936 consensus_queue.cc:260] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.617111 23547 consensus_queue.cc:1048] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20251023 02:05:10.617323 24077 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.657464 23703 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:10.655201 23928 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:10.656314 24133 raft_consensus.cc:2764] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20251023 02:05:10.658675 24144 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:10.756229 22107 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:10.757736 22033 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:10.774919 22032 raft_consensus.cc:2368] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7.
I20251023 02:05:10.705659 24147 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:10.758090 23965 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:10.657382 23717 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.756647 24092 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.678130 23407 consensus_queue.cc:1048] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.112s
I20251023 02:05:10.678673 24043 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:10.775506 23703 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:10.690095 23923 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.681625 23964 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:10.690220 24055 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:10.906948 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 306002us
I20251023 02:05:10.788969 24051 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:10.794101 23936 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:10.755692 23739 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.757210 24107 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.756541 24062 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.831790 24147 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:10.850407 23928 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:10.934732 22107 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251023 02:05:10.690297 24143 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.906024 23407 consensus_queue.cc:1243] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251023 02:05:10.939061 24092 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:10.974740 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 393882us
I20251023 02:05:10.992820 24092 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:10.995810 24080 consensus_queue.cc:1048] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:10.996961 21957 raft_consensus.cc:1275] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:11.001860 21957 raft_consensus.cc:1240] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:10.998075 21955 raft_consensus.cc:1240] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:11.006660 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:11.002645 24086 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:11.012717 22000 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:11.007508 22107 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:11.016250 24086 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:11.020465 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:11.021430 21959 raft_consensus.cc:2410] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251023 02:05:11.021908 21955 raft_consensus.cc:1275] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:11.024637 22000 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:11.024852 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:11.025760 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:11.028496 21955 raft_consensus.cc:1240] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:11.021543 24086 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:11.044677 21908 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f745a2e5ae9f4a829fa01ad2bce79767, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:11.045157 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f745a2e5ae9f4a829fa01ad2bce79767, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.046033 22000 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:11.040113 22107 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:11.058739 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:11.059213 21925 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:11.059345 22107 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:10.965031 24147 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:11.080348 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:11.081007 22037 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:10.921456 23965 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.061143 24158 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:11.103345 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:11.108249 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:11.109031 22033 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:11.077409 23999 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:11.117188 23923 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:11.118599 24086 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:11.118903 23965 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:11.127904 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:11.128738 22037 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:11.143935 24173 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:11.147411 24092 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:11.147764 24086 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:10.676167 22615 rpcz_store.cc:276] 1023 02:05:09.102502 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:09.104772 (+ 2270us) service_pool.cc:225] Handling call
1023 02:05:09.207322 (+102550us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:09.689726 (+482404us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:09.689757 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:09.690240 (+ 483us) log.cc:844] Serialized 26 byte log entry
1023 02:05:10.035728 (+345488us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:10.037075 (+ 1347us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:10.331638 (+294563us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b5400172854. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:10.331694 (+ 56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:10.331743 (+ 49us) raft_consensus.cc:1685] finished
1023 02:05:10.331762 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:10.331999 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":49170,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"spinlock_wait_cycles":336942336,"thread_start_us":601430,"threads_started":2,"wal-append.queue_time_us":438887}
I20251023 02:05:11.148245 24086 raft_consensus.cc:697] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:11.148659 24092 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:11.149067 24055 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:11.144042 24154 log.cc:927] Time spent T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20251023 02:05:11.151237 24143 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:11.126591 24162 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:11.113013 24158 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.151901 24162 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:11.142798 24170 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:11.154150 23964 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:11.128022 24077 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:11.143270 24153 log.cc:927] Time spent T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
W20251023 02:05:11.144438 24155 log.cc:927] Time spent T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.225s user 0.000s sys 0.001s
W20251023 02:05:11.183687 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:11.175439 24111 consensus_meta.cc:287] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.943s user 0.012s sys 0.001s
I20251023 02:05:11.149178 24086 consensus_queue.cc:237] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.156183 24166 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:11.214711 21908 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c0594e36e0824e13965498e412eee3fb, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:11.216856 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c0594e36e0824e13965498e412eee3fb, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.149295 24092 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.154476 24170 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:11.223409 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 177460us
I20251023 02:05:11.157110 24168 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:11.200811 24110 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:11.231009 23928 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:11.214393 23964 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:11.204849 24031 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:11.223940 24162 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:11.260450 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 220944us
I20251023 02:05:11.228091 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:11.217536 24144 raft_consensus.cc:697] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:11.264457 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:11.266944 21959 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:11.230629 23739 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:11.250485 24107 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:11.301079 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:11.269150 24162 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:11.381659 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:11.395581 22615 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
W20251023 02:05:11.436826 24122 consensus_meta.cc:287] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.306s user 0.007s sys 0.002s
I20251023 02:05:11.429493 24162 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.312059 24110 consensus_queue.cc:1059] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.138s
I20251023 02:05:11.228539 24166 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.258729 23964 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.254724 24168 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:11.274507 24184 consensus_queue.cc:1048] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.300s
I20251023 02:05:11.542799 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1491) took 1228 ms. Trace:
I20251023 02:05:11.460544 21959 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
W20251023 02:05:11.523423 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 464274us
I20251023 02:05:11.574379 23936 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:11.695842 22621 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251023 02:05:11.699994 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:11.723109 21962 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:11.699817 22109 rpcz_store.cc:276] 1023 02:05:10.314736 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:10.314824 (+ 88us) service_pool.cc:225] Handling call
1023 02:05:10.315140 (+ 316us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:10.315433 (+ 293us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:10.315461 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:10.315839 (+ 378us) log.cc:844] Serialized 26 byte log entry
1023 02:05:10.623009 (+307170us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:10.675934 (+ 52925us) log.cc:844] Serialized 14 byte log entry
1023 02:05:10.940618 (+264684us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:10.940694 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:11.542513 (+601819us) raft_consensus.cc:1685] finished
1023 02:05:11.542539 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:11.542713 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6798,"prepare.queue_time_us":53576,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":434716,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":38843,"spinlock_wait_cycles":22912,"thread_start_us":553207,"threads_started":2,"wal-append.queue_time_us":588915}
W20251023 02:05:11.561625 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:11.543299 23928 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:11.527817 21908 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 00301d3b4a804bf19bad66c51fe19faa, TS 818907d63f1f4d84ba267ebe6ee837e0 -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.437314 24122 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.333738 24172 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:11.713960 24177 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:11.706946 24158 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:11.712065 24107 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:11.712620 24159 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:11.712816 24162 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:10.241979 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:05:11.736951 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 188) took 2758 ms. Trace:
I20251023 02:05:11.754221 22034 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
W20251023 02:05:11.738083 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 00301d3b4a804bf19bad66c51fe19faa, TS 818907d63f1f4d84ba267ebe6ee837e0 -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.743527 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:11.744645 24092 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:11.714011 24031 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:11.757668 22590 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:11.440912 24152 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:11.503756 24178 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:11.754769 22031 rpcz_store.cc:276] 1023 02:05:08.967713 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:08.967956 (+ 243us) service_pool.cc:225] Handling call
1023 02:05:08.968230 (+ 274us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:09.292299 (+324069us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:09.292335 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:09.292376 (+ 41us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:09.292409 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:11.723390 (+2430981us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b540040d3d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:11.723455 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:11.723625 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":744947072}
W20251023 02:05:11.776352 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 75079.7us
I20251023 02:05:11.777475 21925 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:11.782413 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:11.783012 22109 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:11.752974 23999 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:11.751417 24033 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:11.725131 23936 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.808172 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:11.808239 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:11.725481 24144 consensus_queue.cc:237] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:11.811818 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:11.785518 22623 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 6
I20251023 02:05:11.768271 22590 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:11.253671 24111 raft_consensus.cc:2955] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:11.748430 23928 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.832690 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:11.776181 24031 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.817003 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:11.837730 22111 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:11.838822 24031 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:11.744334 24172 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:11.822206 22108 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.688s user 0.002s sys 0.044s
W20251023 02:05:11.805891 23736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.029s sys 0.037s
I20251023 02:05:11.814576 24086 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20251023 02:05:11.850180 22111 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:11.753926 24107 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:11.851023 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 91479.7us
I20251023 02:05:11.853749 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:11.854734 22108 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s user 0.002s sys 0.044s
I20251023 02:05:11.744365 24177 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:11.814426 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:11.862644 21928 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:11.858955 23736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s user 0.029s sys 0.037s
I20251023 02:05:11.854645 24031 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:11.852447 24170 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:11.814342 23999 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.755923 24159 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:12.006469 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 143917us
I20251023 02:05:11.814200 24174 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:11.951208 24162 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:11.857872 24188 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:11.954180 24184 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:12.006826 24192 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:11.789814 24152 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.004240 24193 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:12.005162 24170 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:12.092759 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 466211us
W20251023 02:05:12.126928 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:12.154567 21908 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4a43ae59c3e347fd86371312be59bb23, TS 02c53ba06cc3445a8534a84aa76b1c14 -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:12.179030 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4a43ae59c3e347fd86371312be59bb23, TS 02c53ba06cc3445a8534a84aa76b1c14 -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.153095 23736 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:12.017165 24174 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:11.993983 24031 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:11.789677 24178 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.041286 24192 raft_consensus.cc:697] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:12.313607 22107 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.245s user 0.001s sys 0.019s
W20251023 02:05:12.361800 22002 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.019s sys 0.011s
W20251023 02:05:12.362282 22002 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.648s user 0.020s sys 0.012s
W20251023 02:05:12.362759 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 774846us
I20251023 02:05:12.364696 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:12.351938 22000 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.005s sys 0.023s
W20251023 02:05:12.366153 22000 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.648s user 0.005s sys 0.023s
W20251023 02:05:12.366504 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 669884us
W20251023 02:05:12.207348 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:12.404027 22107 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.292s user 0.002s sys 0.019s
I20251023 02:05:12.404330 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:12.404902 22587 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.136s user 0.003s sys 0.029s
I20251023 02:05:12.405016 22111 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
W20251023 02:05:12.405272 22587 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.138s user 0.003s sys 0.029s
W20251023 02:05:12.405565 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 1.13885e+06us
I20251023 02:05:12.365588 22109 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:12.036537 24184 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.360836 21928 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:12.359360 22618 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.004s sys 0.010s
I20251023 02:05:12.030637 24188 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.417060 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.420715 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.424446 22109 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 2.
W20251023 02:05:12.425770 24194 log.cc:927] Time spent T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.248s user 0.001s sys 0.000s
I20251023 02:05:12.273475 24092 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:12.201541 23964 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:12.032066 24162 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.429553 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.201988 24174 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:12.418057 24033 consensus_meta.cc:287] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.604s user 0.016s sys 0.003s
I20251023 02:05:12.360508 23965 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:12.403820 23936 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:12.435882 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1379) took 1531 ms. Trace:
I20251023 02:05:12.188865 24193 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:12.419730 22618 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.715s user 0.004s sys 0.011s
I20251023 02:05:12.193691 23736 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.497246 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.517311 22108 rpcz_store.cc:276] 1023 02:05:10.904195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:11.120726 (+216531us) service_pool.cc:225] Handling call
1023 02:05:11.120986 (+ 260us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:11.132923 (+ 11937us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:11.132984 (+ 61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:11.133505 (+ 521us) log.cc:844] Serialized 26 byte log entry
1023 02:05:12.360332 (+1226827us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:12.360381 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:12.360673 (+ 292us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:12.435689 (+ 75016us) raft_consensus.cc:1685] finished
1023 02:05:12.435710 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:12.435832 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11587,"raft.queue_time_us":51036,"raft.run_cpu_time_us":61,"raft.run_wall_time_us":61,"thread_start_us":730042,"threads_started":1,"wal-append.queue_time_us":1198835}
I20251023 02:05:12.192145 24170 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.524456 22111 raft_consensus.cc:1275] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:12.433404 23737 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:12.418186 22615 raft_consensus.cc:1240] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
W20251023 02:05:12.508860 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 531574us
I20251023 02:05:12.431893 24158 raft_consensus.cc:2804] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:12.435480 24093 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:12.522374 22035 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
W20251023 02:05:12.529157 22589 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.006s sys 0.011s
I20251023 02:05:12.509198 23928 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.506147 24086 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:12.535285 22589 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.733s user 0.006s sys 0.012s
I20251023 02:05:12.437707 24092 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.526580 23936 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:12.590183 22590 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.007s sys 0.024s
W20251023 02:05:12.593179 22590 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.757s user 0.007s sys 0.024s
W20251023 02:05:12.593748 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 825580us
W20251023 02:05:12.605561 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:12.559818 24110 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.792s user 0.004s sys 0.016s
W20251023 02:05:12.590819 22619 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.893s user 0.007s sys 0.026s
I20251023 02:05:12.525455 24182 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:12.559473 24144 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.029s sys 0.036s
I20251023 02:05:12.530758 24203 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:12.534943 24112 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:12.554265 24122 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.532938 24168 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:12.618803 22587 connection.cc:232] RPC call timeout handler was delayed by -1.085726912000041s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
W20251023 02:05:12.619375 21960 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.981s user 0.012s sys 0.027s
W20251023 02:05:12.633564 21960 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.996s user 0.013s sys 0.027s
I20251023 02:05:12.561542 23928 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:12.535642 23999 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.523109 23965 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.534430 24107 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:12.611011 24110 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.874s user 0.004s sys 0.016s
I20251023 02:05:12.565944 24187 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:12.618238 24166 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:12.621268 24195 log.cc:927] Time spent T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.572s user 0.001s sys 0.000s
I20251023 02:05:12.639555 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:12.615953 22619 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.919s user 0.007s sys 0.026s
W20251023 02:05:12.621830 22623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.008s sys 0.032s
I20251023 02:05:12.562330 24201 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:12.533358 23737 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:12.627722 24144 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.814s user 0.029s sys 0.037s
I20251023 02:05:12.619942 24188 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:12.656401 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 289156us
W20251023 02:05:12.657444 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 296683us
W20251023 02:05:12.663582 22624 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.011s sys 0.048s
I20251023 02:05:12.664160 24184 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.664703 24178 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:12.665333 24031 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.561542 24086 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.635747 24112 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.667021 23736 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:12.675735 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 73761.7us
I20251023 02:05:12.621999 24177 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:12.682693 22624 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.842s user 0.012s sys 0.048s
I20251023 02:05:12.613346 23936 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.641062 24168 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:12.647872 24158 raft_consensus.cc:697] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:12.645794 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:12.681689 22623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.854s user 0.012s sys 0.036s
I20251023 02:05:12.659415 24110 consensus_queue.cc:1243] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer ca1362b9d9b34073bf35b6f6c9d1159a log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251023 02:05:12.634711 24174 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.560209 24093 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.678124 24107 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:12.708069 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:12.726284 22108 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:12.645635 23928 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:12.564896 24192 consensus_queue.cc:237] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.711349 24223 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election won for term 7
W20251023 02:05:12.711817 24196 log.cc:927] Time spent T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.361s user 0.001s sys 0.000s
W20251023 02:05:12.711711 23703 consensus_meta.cc:287] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.755s user 0.011s sys 0.001s
I20251023 02:05:12.680099 23779 consensus_queue.cc:1059] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20251023 02:05:12.725091 24225 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:12.725250 21925 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:12.727022 24147 consensus_meta.cc:287] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 1.645s user 0.006s sys 0.006s
I20251023 02:05:12.640518 24203 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.720376 24168 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:12.741279 22618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 320) took 1471 ms. Trace:
I20251023 02:05:12.741055 23703 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.702144 24220 consensus_queue.cc:1059] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20251023 02:05:12.698323 24172 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.720611 24144 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:12.732970 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:12.685015 24201 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.759096 22623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 324) took 1034 ms. Trace:
I20251023 02:05:12.766515 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:12.739717 24107 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.677619 24187 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:12.741312 24209 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:12.640383 24182 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.753063 24223 raft_consensus.cc:697] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:12.766846 22623 rpcz_store.cc:276] 1023 02:05:11.724672 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:11.725346 (+ 674us) service_pool.cc:225] Handling call
1023 02:05:11.764067 (+ 38721us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:12.733106 (+969039us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:12.736772 (+ 3666us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:12.739733 (+ 2961us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:12.747184 (+ 7451us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:12.751061 (+ 3877us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:12.757559 (+ 6498us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":840040,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":131,"thread_start_us":893115,"threads_started":1}
I20251023 02:05:12.766798 24203 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.767791 24162 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.751622 24147 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:12.786310 24086 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.786785 23928 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:12.795325 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:12.778582 24092 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.788897 22108 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:12.765632 24170 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:12.761562 24144 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.751106 22618 rpcz_store.cc:276] 1023 02:05:11.295769 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:11.319702 (+ 23933us) service_pool.cc:225] Handling call
1023 02:05:11.703822 (+384120us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:11.704139 (+ 317us) raft_consensus.cc:1534] Early marking committed up to index 5
1023 02:05:12.529252 (+825113us) log.cc:844] Serialized 14 byte log entry
1023 02:05:12.725763 (+196511us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:12.736720 (+ 10957us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:12.739188 (+ 2468us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:12.741101 (+ 1913us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:12.741224 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":923377,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":6282,"raft.queue_time_us":111429,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":74,"spinlock_wait_cycles":7995264,"thread_start_us":977346,"threads_started":3,"wal-append.queue_time_us":99020}
I20251023 02:05:12.795989 22037 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:12.787081 24184 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:12.798039 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:12.797777 24223 consensus_queue.cc:237] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:12.780665 24193 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.783931 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:12.810226 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.813805 22622 raft_consensus.cc:1240] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251023 02:05:12.777318 24158 consensus_queue.cc:237] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:12.808480 22033 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:12.741915 24225 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:12.821738 24144 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:12.802539 24087 log.cc:927] Time spent T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 3.411s user 0.012s sys 0.060s
I20251023 02:05:12.799327 24162 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:12.823725 24093 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.809099 23737 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.796474 23936 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:12.801076 24086 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:12.802145 23703 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.802906 24217 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:12.802445 24191 raft_consensus.cc:2764] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20251023 02:05:12.814419 22000 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:12.837512 22110 raft_consensus.cc:1275] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20251023 02:05:12.840373 22103 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1447) took 4647 ms (client timeout 1665 ms). Trace:
W20251023 02:05:12.846024 22103 rpcz_store.cc:269] 1023 02:05:08.189846 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:08.190163 (+ 317us) service_pool.cc:225] Handling call
1023 02:05:12.836815 (+4646652us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54001abdd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:12.836959 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2822708096}
I20251023 02:05:12.787369 24209 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:12.836304 24107 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.820937 24184 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.852270 24225 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.817204 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.847925 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.844028 24217 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:12.834144 24162 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.841514 24086 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:12.861155 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.863039 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:12.864718 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:12.850626 24093 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:12.850971 24174 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.861918 24086 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:12.837375 24201 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:12.860280 24168 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:12.857364 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.895279 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.905260 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.871834 22615 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:12.909154 22103 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:12.915416 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.911357 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1494) took 2332 ms. Trace:
I20251023 02:05:12.878376 24093 raft_consensus.cc:697] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:12.932394 22108 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:12.932870 24093 consensus_queue.cc:237] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:12.940805 22075 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:12.904475 24187 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:12.933331 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.946861 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.948921 22103 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:12.949029 22108 raft_consensus.cc:2410] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20251023 02:05:12.945523 22107 rpcz_store.cc:276] 1023 02:05:10.579167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:11.067435 (+488268us) service_pool.cc:225] Handling call
1023 02:05:11.068609 (+ 1174us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:12.404481 (+1335872us) raft_consensus.cc:1534] Early marking committed up to index 6
1023 02:05:12.621006 (+216525us) log.cc:844] Serialized 14 byte log entry
1023 02:05:12.910925 (+289919us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:12.910997 (+ 72us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:12.911053 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:12.911141 (+ 88us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:12.911289 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":307742,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":103933,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":10503,"thread_start_us":1825174,"threads_started":3}
I20251023 02:05:12.952296 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.951402 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.953408 22103 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:12.954319 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.945385 24147 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.956480 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.954488 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.949548 24168 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:12.960538 22103 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:12.960649 22107 raft_consensus.cc:2410] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251023 02:05:12.948108 24188 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:12.961215 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.965698 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.955745 24182 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:12.968555 22103 raft_consensus.cc:1240] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
W20251023 02:05:13.146870 21926 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.000s sys 0.031s
I20251023 02:05:13.156876 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1437) took 3732 ms. Trace:
W20251023 02:05:13.157864 21960 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.008s sys 0.025s
W20251023 02:05:13.167963 22034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1151) took 2657 ms (client timeout 1502 ms). Trace:
I20251023 02:05:13.003012 22619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 328) took 1691 ms. Trace:
I20251023 02:05:13.053422 23780 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:13.049021 23965 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:13.097316 24043 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:13.092299 24061 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:13.022975 24062 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:13.021214 24086 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:13.001822 24111 consensus_meta.cc:287] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.962s user 0.007s sys 0.005s
I20251023 02:05:13.332031 22103 raft_consensus.cc:1240] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20251023 02:05:13.076849 24112 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:13.101389 24131 log.cc:927] Time spent T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 3.005s user 0.014s sys 0.025s
W20251023 02:05:13.009406 24139 log.cc:927] Time spent T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 2.749s user 0.010s sys 0.015s
W20251023 02:05:13.060256 24150 log.cc:927] Time spent T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 2.081s user 0.009s sys 0.023s
W20251023 02:05:13.354338 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:13.047084 24152 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:13.355249 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:13.045670 24159 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:13.357820 22104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1488) took 3048 ms. Trace:
I20251023 02:05:12.998214 24162 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:13.041951 24164 log.cc:927] Time spent T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 1.824s user 0.019s sys 0.029s
I20251023 02:05:12.964164 24168 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:13.375029 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:13.375268 24168 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:13.375769 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1492) took 3059 ms. Trace:
I20251023 02:05:13.376729 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:13.086663 24181 log.cc:927] Time spent T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 1.533s user 0.009s sys 0.014s
I20251023 02:05:13.376222 22106 rpcz_store.cc:276] 1023 02:05:10.319598 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:10.320486 (+ 888us) service_pool.cc:225] Handling call
1023 02:05:10.320755 (+ 269us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:10.658271 (+337516us) raft_consensus.cc:1534] Early marking committed up to index 2
1023 02:05:10.658301 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:10.658762 (+ 461us) log.cc:844] Serialized 26 byte log entry
1023 02:05:10.925888 (+267126us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:10.925941 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:10.926034 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:13.041520 (+2115486us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b24003892b8. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c83ffe4 00007f9c5c83e0eb 00007f9c5c8cce49 00007f9c5c8ccad5 00007f9c5c8cc11d 00007f9c5f61920e 00007f9c5f63093c 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1
1023 02:05:13.375251 (+333731us) raft_consensus.cc:1685] finished
1023 02:05:13.375292 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.375606 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":35198,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"spinlock_wait_cycles":8006528,"thread_start_us":603735,"threads_started":2,"wal-append.queue_time_us":558813}
I20251023 02:05:13.358294 22104 rpcz_store.cc:276] 1023 02:05:10.310583 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:10.311250 (+ 667us) service_pool.cc:225] Handling call
1023 02:05:10.311533 (+ 283us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:10.311867 (+ 334us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:10.311892 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:10.312255 (+ 363us) log.cc:844] Serialized 26 byte log entry
1023 02:05:10.613938 (+301683us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:05:10.614771 (+ 833us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:10.614827 (+ 56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:13.059429 (+2444602us) spinlock_profiling.cc:243] Waited 3.39 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:13.357190 (+297761us) raft_consensus.cc:1685] finished
1023 02:05:13.357235 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.357632 (+ 397us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":818,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":7793408,"thread_start_us":301380,"threads_started":1,"wal-append.queue_time_us":591520}
I20251023 02:05:13.378736 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.991133 24184 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:13.379087 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:13.305527 22619 rpcz_store.cc:276] 1023 02:05:11.311693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:11.426917 (+115224us) service_pool.cc:225] Handling call
1023 02:05:11.696547 (+269630us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:11.696771 (+ 224us) raft_consensus.cc:1534] Early marking committed up to index 6
1023 02:05:12.693514 (+996743us) log.cc:844] Serialized 14 byte log entry
1023 02:05:12.966368 (+272854us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:12.967098 (+ 730us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:12.973532 (+ 6434us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:12.981452 (+ 7920us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:12.999455 (+ 18003us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":863885,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":15438,"raft.queue_time_us":1372,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":4121,"thread_start_us":1231585,"threads_started":2,"wal-append.queue_time_us":612412}
I20251023 02:05:13.379987 22104 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:13.380427 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1461) took 2383 ms. Trace:
I20251023 02:05:13.382035 22076 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:13.102279 24189 raft_consensus.cc:3055] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Stepping down as leader of term 6
W20251023 02:05:13.177201 22105 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1359) took 3610 ms (client timeout 1983 ms). Trace:
I20251023 02:05:13.178078 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1474) took 3827 ms. Trace:
I20251023 02:05:13.178738 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1391) took 1321 ms. Trace:
I20251023 02:05:13.051436 24193 raft_consensus.cc:3055] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Stepping down as leader of term 6
W20251023 02:05:13.382896 22105 rpcz_store.cc:269] 1023 02:05:09.567047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:09.597803 (+ 30756us) service_pool.cc:225] Handling call
1023 02:05:13.176721 (+3578918us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b54001aaed4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:13.176957 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":901559040}
I20251023 02:05:13.387367 24193 raft_consensus.cc:740] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:13.388134 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:13.389329 22105 raft_consensus.cc:2410] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251023 02:05:13.055075 24213 raft_consensus.cc:3055] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:12.996435 24217 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:13.395092 24213 raft_consensus.cc:740] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:13.254679 21926 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s user 0.000s sys 0.032s
W20251023 02:05:13.272955 21960 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s user 0.009s sys 0.025s
W20251023 02:05:13.397440 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 775835us
W20251023 02:05:13.238209 22034 rpcz_store.cc:269] 1023 02:05:10.510037 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:10.568219 (+ 58182us) service_pool.cc:225] Handling call
1023 02:05:13.167368 (+2599149us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54001a0c54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:13.167590 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2725606144}
I20251023 02:05:13.237618 21956 rpcz_store.cc:276] 1023 02:05:09.424657 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:09.432539 (+ 7882us) service_pool.cc:225] Handling call
1023 02:05:10.229161 (+796622us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:10.229969 (+ 808us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.156067 (+2926098us) spinlock_profiling.cc:243] Waited 796 ms on lock 0x7b5400218c50. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:13.156510 (+ 443us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1830991744}
W20251023 02:05:13.295040 22077 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.012s sys 0.009s
W20251023 02:05:13.294780 22002 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.012s sys 0.007s
W20251023 02:05:13.309896 24052 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.919s user 0.016s sys 0.061s
I20251023 02:05:13.322472 22037 raft_consensus.cc:1240] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 3. Current term is 4. Ops: []
W20251023 02:05:13.323066 24114 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 3.028s user 0.012s sys 0.023s
W20251023 02:05:13.345853 21925 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.007s sys 0.008s
W20251023 02:05:13.349085 22035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1363) took 3207 ms (client timeout 2066 ms). Trace:
W20251023 02:05:13.333285 24116 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 3.018s user 0.007s sys 0.043s
I20251023 02:05:13.330245 24086 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:13.370675 22000 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:13.382234 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1448) took 3954 ms. Trace:
I20251023 02:05:13.380100 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:13.380870 21958 rpcz_store.cc:276] 1023 02:05:10.996533 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:10.996670 (+ 137us) service_pool.cc:225] Handling call
1023 02:05:10.996954 (+ 284us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:10.997254 (+ 300us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:10.997288 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:10.998361 (+ 1073us) log.cc:844] Serialized 52 byte log entry
1023 02:05:11.017526 (+ 19165us) raft_consensus.cc:1645] Marking committed up to 1
1023 02:05:11.017954 (+ 428us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:11.018385 (+ 431us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:13.085775 (+2067390us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:13.379884 (+294109us) raft_consensus.cc:1685] finished
1023 02:05:13.379917 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.380180 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":140513,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":5835,"spinlock_wait_cycles":10319616,"thread_start_us":18049,"threads_started":1,"wal-append.queue_time_us":262154}
I20251023 02:05:12.967233 24188 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:13.383530 22102 rpcz_store.cc:276] 1023 02:05:11.857339 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:11.859506 (+ 2167us) service_pool.cc:225] Handling call
1023 02:05:11.994457 (+134951us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:12.438233 (+443776us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:12.499156 (+ 60923us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:12.509633 (+ 10477us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:12.514648 (+ 5015us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.178239 (+663591us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b54002fc1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:13.178302 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.178546 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":993067776}
I20251023 02:05:13.396953 24213 consensus_queue.cc:260] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:13.443851 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:13.444990 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 63404.8us
I20251023 02:05:13.382918 24189 raft_consensus.cc:740] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:13.445235 24213 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:05:13.365351 23956 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.428s user 0.021s sys 0.047s
I20251023 02:05:13.372530 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:12.974447 24209 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:13.383862 22110 rpcz_store.cc:276] 1023 02:05:09.350951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:09.449976 (+ 99025us) service_pool.cc:225] Handling call
1023 02:05:12.837081 (+3387105us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:12.845628 (+ 8547us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.177615 (+331987us) spinlock_profiling.cc:243] Waited 3.39 s on lock 0x7b54001abdd0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:13.177888 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7789475584}
I20251023 02:05:13.389106 24193 consensus_queue.cc:260] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:13.463307 22110 raft_consensus.cc:1240] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20251023 02:05:13.464087 22036 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251023 02:05:13.383560 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:13.466292 24189 consensus_queue.cc:260] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:13.394217 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.386004 22109 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
W20251023 02:05:13.413421 22032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1154) took 2902 ms (client timeout 2000 ms). Trace:
W20251023 02:05:13.416383 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:13.417023 22077 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.802s user 0.013s sys 0.009s
W20251023 02:05:13.406176 21925 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.654s user 0.007s sys 0.009s
I20251023 02:05:13.406370 21954 rpcz_store.cc:276] 1023 02:05:09.427657 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:09.463974 (+ 36317us) service_pool.cc:225] Handling call
1023 02:05:09.464360 (+ 386us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:09.620289 (+155929us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:09.620332 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:09.621027 (+ 695us) log.cc:844] Serialized 52 byte log entry
1023 02:05:09.983197 (+362170us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:09.983242 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:09.983328 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:13.100856 (+3117528us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b24000ed0f8. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c83ffe4 00007f9c5c83e0eb 00007f9c5c8cce49 00007f9c5c8ccad5 00007f9c5c8cc11d 00007f9c5f61920e 00007f9c5f63093c 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1
1023 02:05:13.381664 (+280808us) raft_consensus.cc:1685] finished
1023 02:05:13.381706 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.382021 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78000,"raft.queue_time_us":79047,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":6332,"spinlock_wait_cycles":3682560,"thread_start_us":318312,"threads_started":1,"wal-append.queue_time_us":407800}
W20251023 02:05:13.432502 22002 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.069s user 0.012s sys 0.007s
W20251023 02:05:13.417615 24116 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.142s user 0.008s sys 0.044s
W20251023 02:05:13.400640 22035 rpcz_store.cc:269] 1023 02:05:10.141763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:10.141929 (+ 166us) service_pool.cc:225] Handling call
1023 02:05:13.348588 (+3206659us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b54002a94d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:13.348864 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5472831872}
I20251023 02:05:13.381187 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:13.445443 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:13.428413 24114 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.161s user 0.013s sys 0.024s
I20251023 02:05:13.455098 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:13.456061 23964 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:13.457870 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:13.459044 22624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 329) took 2031 ms. Trace:
I20251023 02:05:13.460176 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.445909 24086 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20251023 02:05:13.461201 22622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 225) took 2473 ms. Trace:
W20251023 02:05:13.422585 24052 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.050s user 0.016s sys 0.062s
I20251023 02:05:13.467814 24189 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:13.469134 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 98914.5us
I20251023 02:05:13.496999 22621 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:13.466986 22107 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:13.520447 23717 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:13.490387 22622 rpcz_store.cc:276] 1023 02:05:10.989019 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:10.991892 (+ 2873us) service_pool.cc:225] Handling call
1023 02:05:11.074799 (+ 82907us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:12.816000 (+1741201us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.460566 (+644566us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b5400485154. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:13.461012 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3999098752}
I20251023 02:05:13.517567 24093 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:13.520555 24193 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:13.487520 22032 rpcz_store.cc:269] 1023 02:05:10.510895 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:10.583633 (+ 72738us) service_pool.cc:225] Handling call
1023 02:05:13.413014 (+2829381us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54002a80d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:13.413253 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":413439872}
I20251023 02:05:13.487897 23964 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:13.488075 22624 rpcz_store.cc:276] 1023 02:05:11.427525 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:11.668690 (+241165us) service_pool.cc:225] Handling call
1023 02:05:11.748935 (+ 80245us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:11.840422 (+ 91487us) raft_consensus.cc:1534] Early marking committed up to index 6
1023 02:05:12.706084 (+865662us) log.cc:844] Serialized 14 byte log entry
1023 02:05:12.896763 (+190679us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:12.896836 (+ 73us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:12.896896 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.458597 (+561701us) spinlock_profiling.cc:243] Waited 91.2 ms on lock 0x7b54004f1d54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:13.458663 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.458850 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30280,"prepare.queue_time_us":802123,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":654,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":9309,"spinlock_wait_cycles":209745792,"thread_start_us":1054368,"threads_started":2,"wal-append.queue_time_us":258256}
W20251023 02:05:13.518155 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 766590us
W20251023 02:05:13.522477 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 60375.9us
I20251023 02:05:13.503299 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:13.527827 22621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34806 (request call id 323) took 2132 ms (client timeout 1710 ms). Trace:
I20251023 02:05:13.528230 22105 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:13.469839 24158 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:13.543357 24158 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:13.543962 22621 rpcz_store.cc:269] 1023 02:05:11.716534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:11.725966 (+ 9432us) service_pool.cc:225] Handling call
1023 02:05:13.457472 (+1731506us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b5800214598. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:05:13.527711 (+ 70239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31785216}
W20251023 02:05:13.504256 23956 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.602s user 0.022s sys 0.048s
W20251023 02:05:13.538453 24254 log.cc:927] Time spent T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20251023 02:05:13.556221 23911 consensus_queue.cc:1059] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:05:13.557677 23911 raft_consensus.cc:3055] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:13.557966 22102 raft_consensus.cc:1240] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:13.558189 23911 raft_consensus.cc:740] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
W20251023 02:05:13.547976 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 1.17838e+06us
I20251023 02:05:13.552960 23965 consensus_queue.cc:1059] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251023 02:05:13.553920 24158 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:13.542044 23964 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:13.554746 24203 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:13.556439 22000 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:13.545456 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 49985.2us
I20251023 02:05:13.550961 21925 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:13.540475 24093 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:13.594251 24146 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:13.595106 24188 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:13.597512 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.598453 22037 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:13.559895 23911 consensus_queue.cc:260] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:13.602583 23911 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:13.603464 23407 consensus_queue.cc:1048] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:05:13.605973 23993 consensus_queue.cc:1059] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:05:13.606184 23999 consensus_queue.cc:1059] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:05:13.607308 23965 raft_consensus.cc:3055] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:13.616648 23999 raft_consensus.cc:3055] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:13.617372 23999 raft_consensus.cc:740] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:13.616725 23965 raft_consensus.cc:740] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:13.606014 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.624780 23965 consensus_queue.cc:260] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:13.625986 24203 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:13.626662 23965 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:13.629874 24203 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:13.630805 24202 raft_consensus.cc:3055] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:13.631255 24202 raft_consensus.cc:740] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:13.631320 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:13.632541 24202 consensus_queue.cc:260] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:13.633621 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:13.633947 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:13.643654 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 101911us
I20251023 02:05:13.641865 22619 raft_consensus.cc:2393] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:13.642935 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:13.649024 22621 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
W20251023 02:05:13.651750 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 100985us
I20251023 02:05:13.652855 23774 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:13.653404 22623 pending_rounds.cc:85] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting all ops after (but not including) 4
I20251023 02:05:13.653146 21925 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:13.653666 23774 raft_consensus.cc:697] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:13.654423 23996 raft_consensus.cc:2804] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:05:13.653784 22623 pending_rounds.cc:107] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 4.5
I20251023 02:05:13.587229 24093 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:13.657161 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:13.656837 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1462) took 2660 ms. Trace:
I20251023 02:05:13.604867 22002 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:13.675760 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.675413 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.663635 22623 raft_consensus.cc:2917] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Aborting config change with OpId 4.5: Aborted: Op aborted by new leader
I20251023 02:05:13.680109 22000 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:13.673213 24168 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:13.684274 24168 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:13.672966 23996 raft_consensus.cc:697] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:13.661113 21960 rpcz_store.cc:276] 1023 02:05:10.996934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:11.001105 (+ 4171us) service_pool.cc:225] Handling call
1023 02:05:11.001533 (+ 428us) raft_consensus.cc:1471] Updating replica for 3 ops
1023 02:05:11.174532 (+172999us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:11.174564 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
1023 02:05:11.175213 (+ 649us) log.cc:844] Serialized 78 byte log entry
1023 02:05:11.541873 (+366660us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:12.634106 (+1092233us) log.cc:844] Serialized 14 byte log entry
1023 02:05:13.615594 (+981488us) log.cc:844] Serialized 14 byte log entry
1023 02:05:13.654337 (+ 38743us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.654456 (+ 119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:13.654526 (+ 70us) raft_consensus.cc:1685] finished
1023 02:05:13.654542 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.656775 (+ 2233us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4160,"prepare.queue_time_us":1347013,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":109,"raft.queue_time_us":215465,"raft.run_cpu_time_us":415,"raft.run_wall_time_us":12513,"thread_start_us":2435360,"threads_started":5,"wal-append.queue_time_us":683719}
I20251023 02:05:13.692682 22035 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:13.694693 22075 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:13.696069 24245 log.cc:927] Time spent T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.330s user 0.000s sys 0.001s
I20251023 02:05:13.696841 24168 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:13.697432 24168 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:13.698006 24168 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:13.699067 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 46076.7us
I20251023 02:05:13.699950 21925 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:13.701321 21925 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:13.678534 22617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 325) took 2664 ms. Trace:
I20251023 02:05:13.709012 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:13.710593 24249 log.cc:927] Time spent T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.374s user 0.002s sys 0.003s
I20251023 02:05:13.710759 23816 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:13.711495 24189 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:13.711333 24239 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:13.712086 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 46924.5us
I20251023 02:05:13.717443 22618 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20251023 02:05:13.728353 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:13.717789 24202 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:13.718575 24144 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251023 02:05:13.709354 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 49486us
I20251023 02:05:13.738221 24217 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:13.725284 24189 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:13.748116 21961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1451) took 3968 ms (client timeout 2043 ms). Trace:
I20251023 02:05:13.750854 24217 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:13.756415 23996 consensus_queue.cc:237] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:13.750903 21961 rpcz_store.cc:269] 1023 02:05:09.779942 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:09.780108 (+ 166us) service_pool.cc:225] Handling call
1023 02:05:13.720448 (+3940340us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b540024bdd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:13.747848 (+ 27400us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1098894080}
I20251023 02:05:13.725152 23828 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:13.755164 22624 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:13.758633 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.741039 24052 raft_consensus.cc:2804] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:13.740648 24086 raft_consensus.cc:2804] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:13.758441 24217 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:13.734719 24144 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:13.687538 23774 consensus_queue.cc:237] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:13.751509 24189 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:13.751955 22623 raft_consensus.cc:2889] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20251023 02:05:13.768886 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 67650.3us
I20251023 02:05:13.772189 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.772572 22035 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:13.773173 22036 raft_consensus.cc:2410] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251023 02:05:13.777012 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:13.777653 22616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 224) took 3190 ms. Trace:
I20251023 02:05:13.777920 24188 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251023 02:05:13.778010 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.764946 24086 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:13.785596 24086 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:13.786334 24146 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:13.786805 24146 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:13.782126 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.789736 21962 raft_consensus.cc:1240] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:13.764111 24052 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:13.781956 23828 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:13.791836 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1156) took 1003 ms. Trace:
I20251023 02:05:13.791947 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:13.768594 24144 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:13.748965 23999 consensus_queue.cc:260] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:13.810734 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1500) took 1026 ms. Trace:
I20251023 02:05:13.814963 22000 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:13.802536 24052 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:13.801978 22029 rpcz_store.cc:276] 1023 02:05:12.788103 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:12.789784 (+ 1681us) service_pool.cc:225] Handling call
1023 02:05:12.790081 (+ 297us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:12.864298 (+ 74217us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:05:13.386662 (+522364us) log.cc:844] Serialized 14 byte log entry
1023 02:05:13.674760 (+288098us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:13.676905 (+ 2145us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:13.680756 (+ 3851us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.789422 (+108666us) spinlock_profiling.cc:243] Waited 74 ms on lock 0x7b540026fa54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:13.789484 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.791676 (+ 2192us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":689449,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":79,"raft.queue_time_us":3492,"spinlock_wait_cycles":186859136,"thread_start_us":664907,"threads_started":2}
I20251023 02:05:13.766031 24239 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:13.828681 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.829790 22031 raft_consensus.cc:2393] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:13.830451 24239 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:13.840430 23999 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:13.842059 23828 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:13.845707 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:13.816692 22108 rpcz_store.cc:276] 1023 02:05:12.694934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:12.966297 (+271363us) service_pool.cc:225] Handling call
1023 02:05:12.970555 (+ 4258us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:12.971016 (+ 461us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:12.971060 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:12.971747 (+ 687us) log.cc:844] Serialized 26 byte log entry
1023 02:05:13.322778 (+351031us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:13.322965 (+ 187us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.323097 (+ 132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:13.695247 (+372150us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:13.701662 (+ 6415us) raft_consensus.cc:1685] finished
1023 02:05:13.701715 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.721055 (+ 19340us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":137,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":2019,"spinlock_wait_cycles":36329728,"thread_start_us":321851,"threads_started":1,"wal-append.queue_time_us":339822}
I20251023 02:05:13.848927 22624 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
W20251023 02:05:13.850243 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 44232.6us
I20251023 02:05:13.855427 22075 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:13.858175 24112 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:13.858724 23717 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:13.859328 23717 raft_consensus.cc:697] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:13.860473 23717 consensus_queue.cc:237] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:13.864876 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 260140us
I20251023 02:05:13.858657 24112 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:13.866861 24112 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:13.868870 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 197) took 1021 ms. Trace:
I20251023 02:05:13.720064 22617 rpcz_store.cc:276] 1023 02:05:11.008747 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:11.009082 (+ 335us) service_pool.cc:225] Handling call
1023 02:05:11.009482 (+ 400us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:11.009801 (+ 319us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:11.009844 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:13.179777 (+2169933us) spinlock_profiling.cc:243] Waited 55.6 ms on lock 0x7b2c0004b768. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5e3949b1 00007f9c5e3a3fe2 00007f9c5fad4701 00007f9c5fa98fda 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22
1023 02:05:13.181121 (+ 1344us) log.cc:844] Serialized 26 byte log entry
1023 02:05:13.393262 (+212141us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:13.395605 (+ 2343us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.396179 (+ 574us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:13.654639 (+258460us) raft_consensus.cc:1685] finished
1023 02:05:13.654666 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.672756 (+ 18090us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":127854976,"thread_start_us":138641,"threads_started":1,"wal-append.queue_time_us":428517}
I20251023 02:05:13.871132 22103 rpcz_store.cc:276] 1023 02:05:12.847639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:13.467149 (+619510us) service_pool.cc:225] Handling call
1023 02:05:13.467707 (+ 558us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:13.468147 (+ 440us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:13.468177 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:13.539836 (+ 71659us) log.cc:844] Serialized 26 byte log entry
1023 02:05:13.659093 (+119257us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:13.659135 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.659200 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:13.868636 (+209436us) raft_consensus.cc:1685] finished
1023 02:05:13.868658 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.868813 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":118778,"threads_started":1,"wal-append.queue_time_us":326736}
I20251023 02:05:13.866878 23888 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:13.873255 22000 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:13.891093 24093 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:13.892248 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.898048 24092 consensus_queue.cc:1048] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20251023 02:05:13.903700 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:13.904446 23828 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251023 02:05:13.906031 23828 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:13.906576 23828 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:13.906059 22587 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:13.911237 22587 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:13.778096 22616 rpcz_store.cc:276] 1023 02:05:10.587409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:10.593447 (+ 6038us) service_pool.cc:225] Handling call
1023 02:05:10.630726 (+ 37279us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:11.014690 (+383964us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:11.014735 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:13.413917 (+2399182us) spinlock_profiling.cc:243] Waited 50.9 ms on lock 0x7b2c0004b768. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5e3949b1 00007f9c5e3a3fe2 00007f9c5fad4701 00007f9c5fa98fda 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22
1023 02:05:13.414853 (+ 936us) log.cc:844] Serialized 26 byte log entry
1023 02:05:13.588940 (+174087us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:13.593683 (+ 4743us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.593832 (+ 149us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:13.777152 (+183320us) raft_consensus.cc:1685] finished
1023 02:05:13.777175 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.777459 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16919,"raft.queue_time_us":410445,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":23080,"spinlock_wait_cycles":117063552,"thread_start_us":387512,"threads_started":2,"wal-append.queue_time_us":360070}
I20251023 02:05:13.914556 24003 raft_consensus.cc:2749] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:13.913980 24116 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:13.920441 22028 raft_consensus.cc:3055] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:13.924850 22028 raft_consensus.cc:740] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:13.929724 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 66083.5us
W20251023 02:05:13.930680 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:13.933879 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.943749 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1517) took 1131 ms. Trace:
I20251023 02:05:13.944541 22028 consensus_queue.cc:260] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:13.945549 24093 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:13.945654 22028 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:13.947042 24086 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:13.958907 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:13.917618 23888 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:13.969431 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 153) took 2112 ms. Trace:
I20251023 02:05:13.970001 21962 rpcz_store.cc:276] 1023 02:05:11.857405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:11.857601 (+ 196us) service_pool.cc:225] Handling call
1023 02:05:13.750901 (+1893300us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:13.811331 (+ 60430us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.969118 (+157787us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b540024bdd0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:13.969308 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4293425920}
I20251023 02:05:13.951766 24086 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:13.944204 22110 rpcz_store.cc:276] 1023 02:05:12.812545 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:13.464223 (+651678us) service_pool.cc:225] Handling call
1023 02:05:13.523995 (+ 59772us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:13.528811 (+ 4816us) raft_consensus.cc:1534] Early marking committed up to index 6
1023 02:05:13.529164 (+ 353us) log.cc:844] Serialized 14 byte log entry
1023 02:05:13.874284 (+345120us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:13.874358 (+ 74us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:13.874439 (+ 81us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:13.874529 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:13.943567 (+ 69038us) spinlock_profiling.cc:243] Waited 59.4 ms on lock 0x7b54002d05d0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:13.943692 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3415,"prepare.queue_time_us":11938,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":10253,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":16550,"spinlock_wait_cycles":136589184,"thread_start_us":336598,"threads_started":1}
I20251023 02:05:14.022460 23888 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:14.038062 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 126910us
W20251023 02:05:14.036726 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 355471us
I20251023 02:05:13.964398 23828 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:14.044279 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:14.044445 22077 leader_election.cc:336] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.869s (SENT)
I20251023 02:05:14.046175 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:14.048602 22621 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251023 02:05:14.049068 24069 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:14.050812 22028 raft_consensus.cc:1275] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20251023 02:05:14.051323 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:14.042992 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:14.052615 22028 pending_rounds.cc:85] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting all ops after (but not including) 2
I20251023 02:05:14.058064 22028 pending_rounds.cc:107] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251023 02:05:14.058496 22028 raft_consensus.cc:2889] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251023 02:05:14.060137 24265 log.cc:927] Time spent T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20251023 02:05:14.052170 24069 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:14.077840 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:14.080197 22587 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:14.055516 24168 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:14.094369 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 49992.7us
I20251023 02:05:14.095052 22028 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:14.078547 24069 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:14.096313 24156 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20251023 02:05:14.101984 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:14.101265 22077 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 2.000s (SENT) [suppressed 12 similar messages]
W20251023 02:05:14.108980 24266 log.cc:927] Time spent T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20251023 02:05:14.102249 23888 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:14.105085 24052 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:14.119267 21856 catalog_manager.cc:5649] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 3 to 5, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 5 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:14.098299 24189 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:14.130856 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:14.114686 24168 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:14.134719 22034 raft_consensus.cc:2393] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:14.135447 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:14.150511 22032 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:14.142350 21856 catalog_manager.cc:5649] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 2 to 5, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 5 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:14.159972 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:14.151525 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:14.167716 22002 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:14.186682 24168 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:14.187570 24168 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:14.162621 21856 catalog_manager.cc:5649] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 3 to 4, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 4 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:14.193038 23816 raft_consensus.cc:2749] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:14.196628 22000 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:14.206429 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:14.200040 23965 consensus_meta.cc:287] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.573s user 0.009s sys 0.005s
W20251023 02:05:14.253631 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:14.272169 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 64425.5us
I20251023 02:05:14.276531 21856 catalog_manager.cc:5649] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 3 to 4, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 4 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
W20251023 02:05:14.299911 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 144909us
I20251023 02:05:14.185434 24112 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:14.301172 22622 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.019s sys 0.041s
W20251023 02:05:14.302452 22622 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s user 0.019s sys 0.041s
I20251023 02:05:14.212350 24213 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:14.325701 24186 consensus_queue.cc:1048] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20251023 02:05:14.331365 22029 raft_consensus.cc:2393] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:14.352332 22028 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:14.354908 22034 raft_consensus.cc:1275] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251023 02:05:14.324537 24213 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:14.299180 23965 consensus_queue.cc:1048] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.046s
I20251023 02:05:14.375883 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:14.370645 23964 consensus_meta.cc:287] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.724s user 0.008s sys 0.005s
W20251023 02:05:14.391845 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:14.370995 24052 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:14.379572 24213 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:14.390792 24188 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.008s sys 0.044s
W20251023 02:05:14.408118 21859 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56864 (request call id 222) took 15395 ms (client timeout 15000 ms). Trace:
I20251023 02:05:14.360543 21856 catalog_manager.cc:5649] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: config changed from index 2 to 3, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1), VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2), VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3), VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 3 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
W20251023 02:05:14.415844 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 85253.4us
I20251023 02:05:14.424507 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:14.410272 24188 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s user 0.008s sys 0.044s
I20251023 02:05:14.422683 22000 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:14.411090 24111 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.023s sys 0.032s
I20251023 02:05:14.407567 24168 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:14.372618 24283 consensus_queue.cc:1048] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20251023 02:05:14.412529 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:14.416632 24189 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:14.396503 23964 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:14.431787 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:14.429874 24111 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.084s user 0.023s sys 0.033s
I20251023 02:05:14.438795 21856 catalog_manager.cc:5649] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 3 to 5, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 5 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
W20251023 02:05:14.446141 21925 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.011s sys 0.010s
W20251023 02:05:14.451365 21925 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s user 0.011s sys 0.010s
W20251023 02:05:14.454528 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 554887us
I20251023 02:05:14.455138 21856 catalog_manager.cc:5649] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 3 to 5, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 5 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:14.457957 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:14.462555 21856 catalog_manager.cc:5649] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
W20251023 02:05:14.450435 22623 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.020s sys 0.049s
W20251023 02:05:14.446466 24193 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.009s sys 0.012s
I20251023 02:05:14.470398 22028 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
W20251023 02:05:14.451948 23774 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.004s sys 0.019s
W20251023 02:05:14.457751 23717 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.011s sys 0.012s
I20251023 02:05:14.461241 24186 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:14.446020 24168 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:05:14.472025 24193 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s user 0.010s sys 0.012s
W20251023 02:05:14.475893 23774 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.004s sys 0.020s
W20251023 02:05:14.479658 23717 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s user 0.011s sys 0.012s
I20251023 02:05:14.503434 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:14.504436 22103 raft_consensus.cc:2410] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20251023 02:05:14.505460 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:14.506220 22108 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:14.507121 23774 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:14.507678 23774 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:14.476394 24112 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:14.488229 21856 catalog_manager.cc:5649] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:14.513607 22029 raft_consensus.cc:1275] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:05:14.514997 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:14.510018 23774 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:14.520696 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:14.483340 24217 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.003s sys 0.025s
I20251023 02:05:14.484513 24186 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:14.511639 24146 consensus_meta.cc:287] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.724s user 0.014s sys 0.001s
I20251023 02:05:14.526669 22104 raft_consensus.cc:1240] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
W20251023 02:05:14.525172 24217 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s user 0.003s sys 0.026s
I20251023 02:05:14.530869 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:14.531920 22029 raft_consensus.cc:2393] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
W20251023 02:05:14.515609 23755 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.013s sys 0.016s
I20251023 02:05:14.541079 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:14.519655 22623 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s user 0.021s sys 0.049s
I20251023 02:05:14.545776 22107 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 5
I20251023 02:05:14.528762 24186 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:14.537222 23755 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s user 0.014s sys 0.016s
I20251023 02:05:14.562757 22029 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20251023 02:05:14.410786 21859 rpcz_store.cc:269] 1023 02:04:58.796535 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:04:58.802006 (+ 5471us) service_pool.cc:225] Handling call
1023 02:05:14.172973 (+15370967us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:05:07.793334 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:05:07.794034 (+ 700us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:05:07.794092 (+ 58us) write_op.cc:435] Acquired schema lock
1023 02:05:07.794111 (+ 19us) tablet.cc:660] Decoding operations
1023 02:05:07.797938 (+ 3827us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:05:07.798037 (+ 99us) write_op.cc:641] Partition lock acquired for write op
1023 02:05:07.798069 (+ 32us) tablet.cc:683] Acquiring locks for 10 operations
1023 02:05:07.798942 (+ 873us) tablet.cc:699] Row locks acquired
1023 02:05:07.798961 (+ 19us) write_op.cc:260] PREPARE: finished
1023 02:05:07.799117 (+ 156us) write_op.cc:270] Start()
1023 02:05:07.799275 (+ 158us) write_op.cc:276] Timestamp: P: 1761185107799047 usec, L: 0
1023 02:05:07.799309 (+ 34us) op_driver.cc:348] REPLICATION: starting
1023 02:05:07.896248 (+ 96939us) log.cc:844] Serialized 3973 byte log entry
1023 02:05:08.084635 (+188387us) op_driver.cc:464] REPLICATION: finished
1023 02:05:08.545989 (+461354us) spinlock_profiling.cc:243] Waited 3.91 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5c9cdfdc 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c634c7650
1023 02:05:08.860017 (+314028us) write_op.cc:301] APPLY: starting
1023 02:05:08.860108 (+ 91us) tablet.cc:1365] starting BulkCheckPresence
1023 02:05:08.860370 (+ 262us) tablet.cc:1368] finished BulkCheckPresence
1023 02:05:08.860407 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:05:09.148630 (+288223us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:05:09.317547 (+168917us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
1023 02:05:09.321851 (+ 4304us) write_op.cc:312] APPLY: finished
1023 02:05:09.322196 (+ 345us) log.cc:844] Serialized 76 byte log entry
1023 02:05:09.434618 (+112422us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:05:09.443527 (+ 8909us) write_op.cc:454] Released schema lock
1023 02:05:09.457806 (+ 14279us) write_op.cc:341] FINISH: Updating metrics
1023 02:05:09.458259 (+ 453us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:05:09.458290 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1337,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":758303,"apply.run_cpu_time_us":18434,"apply.run_wall_time_us":598244,"num_ops":10,"prepare.queue_time_us":500475,"prepare.run_cpu_time_us":38620,"prepare.run_wall_time_us":220702,"raft.queue_time_us":13571,"raft.run_cpu_time_us":17817,"raft.run_wall_time_us":693868,"replication_time_us":285150,"spinlock_wait_cycles":8992640,"thread_start_us":1291737,"threads_started":5,"wal-append.queue_time_us":305525,"wal-append.run_cpu_time_us":10711,"wal-append.run_wall_time_us":2310013}]]}
I20251023 02:05:14.548146 21856 catalog_manager.cc:5649] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:14.533146 24168 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:14.530325 24146 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:14.579573 22103 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:14.609936 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:14.608183 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:14.613677 22002 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:14.615612 21856 catalog_manager.cc:5649] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:14.621261 21960 raft_consensus.cc:1240] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:14.613325 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:14.618090 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:14.629283 23888 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251023 02:05:14.629691 24189 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:14.633766 21962 raft_consensus.cc:2410] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251023 02:05:14.636878 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:14.607651 22617 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.017s sys 0.053s
I20251023 02:05:14.613134 23964 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:14.638307 21856 catalog_manager.cc:5649] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 3 to 4, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 4 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:14.651711 22590 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:14.640092 22617 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s user 0.018s sys 0.054s
I20251023 02:05:14.677242 21856 catalog_manager.cc:5649] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 3 to 4, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:14.653244 22618 raft_consensus.cc:2393] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
W20251023 02:05:14.702872 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 454523us
W20251023 02:05:14.711696 24271 log.cc:927] Time spent T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20251023 02:05:14.713562 24146 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:14.714672 22622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 332) took 1082 ms. Trace:
W20251023 02:05:14.711802 24272 log.cc:927] Time spent T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.162s user 0.000s sys 0.000s
I20251023 02:05:14.632735 22032 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:14.715106 22622 rpcz_store.cc:276] 1023 02:05:13.632689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:13.642367 (+ 9678us) service_pool.cc:225] Handling call
1023 02:05:13.661844 (+ 19477us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:13.704368 (+ 42524us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:13.704426 (+ 58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:13.740583 (+ 36157us) log.cc:844] Serialized 26 byte log entry
1023 02:05:14.304157 (+563574us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:14.304213 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:14.304310 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:14.714402 (+410092us) raft_consensus.cc:1685] finished
1023 02:05:14.714427 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:14.714598 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4192,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":3881,"thread_start_us":545719,"threads_started":1,"wal-append.queue_time_us":768028}
I20251023 02:05:14.722064 22032 raft_consensus.cc:606] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:14.743456 21962 raft_consensus.cc:1275] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251023 02:05:14.745432 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 146403us
I20251023 02:05:14.748368 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:14.715170 24189 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:14.741613 21856 catalog_manager.cc:5649] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 2, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 2 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:14.789902 22107 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:14.807330 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:14.810379 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 158772us
I20251023 02:05:14.793062 21856 catalog_manager.cc:5649] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:14.742264 24146 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:14.819456 21856 catalog_manager.cc:5649] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:14.835851 21954 raft_consensus.cc:1240] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
W20251023 02:05:14.839074 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:14.823825 21961 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
W20251023 02:05:14.841202 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:14.847842 24275 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:14.848495 24275 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:14.849099 22618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34806 (request call id 335) took 1058 ms. Trace:
W20251023 02:05:14.849450 22062 heartbeater.cc:646] Failed to heartbeat to 127.20.81.62:41229 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.20.81.62:41229 timed out after 15.000s (SENT)
I20251023 02:05:14.849676 22618 rpcz_store.cc:276] 1023 02:05:13.791050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:13.791633 (+ 583us) service_pool.cc:225] Handling call
1023 02:05:14.848831 (+1057198us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b540059ced4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:14.849022 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1868123392}
I20251023 02:05:14.850904 24275 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:14.845939 24193 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:14.852101 24193 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
W20251023 02:05:14.852739 24273 log.cc:927] Time spent T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
W20251023 02:05:14.852063 24282 log.cc:927] Time spent T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20251023 02:05:14.854595 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:14.855525 24193 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:14.858244 24144 consensus_meta.cc:287] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.037s user 0.014s sys 0.000s
I20251023 02:05:14.793285 24189 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:14.870256 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:14.871297 21961 raft_consensus.cc:2410] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20251023 02:05:14.867175 24069 consensus_meta.cc:287] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.759s user 0.012s sys 0.001s
I20251023 02:05:14.858666 24144 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:14.865008 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:14.894794 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:14.895586 21961 raft_consensus.cc:2393] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:14.896375 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:14.896678 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:14.897215 21959 raft_consensus.cc:2368] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7.
I20251023 02:05:14.896395 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:14.903679 21928 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:14.878306 24168 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:14.908783 21960 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:14.910096 21856 catalog_manager.cc:5649] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 4, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 4 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:14.905609 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:14.899696 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:14.912156 21956 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 3.
I20251023 02:05:14.912166 21955 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:14.922051 21954 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:14.897563 22623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 333) took 1264 ms. Trace:
I20251023 02:05:14.920629 21856 catalog_manager.cc:5649] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 5, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 5 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
W20251023 02:05:14.937474 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 58876.3us
W20251023 02:05:14.942463 22076 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.016s sys 0.010s
W20251023 02:05:14.942832 22076 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.570s user 0.016s sys 0.010s
I20251023 02:05:14.943284 22108 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
W20251023 02:05:14.945072 22075 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.012s sys 0.014s
W20251023 02:05:14.945459 22075 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.012s sys 0.015s
I20251023 02:05:14.892664 24069 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:14.951192 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:14.966033 22032 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
W20251023 02:05:14.968883 22587 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.009s sys 0.020s
W20251023 02:05:14.970980 22587 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.587s user 0.009s sys 0.020s
I20251023 02:05:14.923305 24193 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:14.978435 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:14.980433 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:14.966961 22032 raft_consensus.cc:606] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:14.981488 22077 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.013s sys 0.018s
I20251023 02:05:14.965910 21856 catalog_manager.cc:5649] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a reported cstate change: term changed from 1 to 4, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
W20251023 02:05:14.993099 22077 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.014s sys 0.018s
W20251023 02:05:14.993489 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 603231us
I20251023 02:05:14.993631 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:14.994437 21955 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:14.992627 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:14.991291 21928 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:14.997484 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:15.000229 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.001299 22028 raft_consensus.cc:2368] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5.
I20251023 02:05:15.009156 22108 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:15.020336 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:14.983949 24193 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
W20251023 02:05:15.007941 22076 connection.cc:232] RPC call timeout handler was delayed by -3.0691977699999597s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1502s for a response
I20251023 02:05:15.023967 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:15.026928 22076 connection.cc:232] RPC call timeout handler was delayed by -3.0520977699999321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1521s for a response
W20251023 02:05:15.027383 22076 connection.cc:232] RPC call timeout handler was delayed by -3.04849777000004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
W20251023 02:05:15.027746 22076 connection.cc:232] RPC call timeout handler was delayed by -2.9755977700000358s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
W20251023 02:05:15.028098 22076 connection.cc:232] RPC call timeout handler was delayed by -2.8765977699999894s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
W20251023 02:05:15.028492 22076 connection.cc:232] RPC call timeout handler was delayed by -2.8576977699999588s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
W20251023 02:05:15.028872 22076 connection.cc:232] RPC call timeout handler was delayed by -2.8549977699999545s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
W20251023 02:05:15.029217 22076 connection.cc:232] RPC call timeout handler was delayed by -2.8216977700000143s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1777s for a response
I20251023 02:05:15.024825 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.030704 22105 raft_consensus.cc:2410] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251023 02:05:14.996755 24168 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
W20251023 02:05:15.031968 22589 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.011s sys 0.027s
I20251023 02:05:15.032070 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:15.034603 22589 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.624s user 0.011s sys 0.027s
I20251023 02:05:15.033509 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:15.034941 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 664726us
I20251023 02:05:15.040469 22028 raft_consensus.cc:2393] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:05:15.029569 22076 connection.cc:232] RPC call timeout handler was delayed by -2.8162977700000056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
W20251023 02:05:15.048484 22076 connection.cc:232] RPC call timeout handler was delayed by -2.7514977700000145s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
W20251023 02:05:15.048919 22076 connection.cc:232] RPC call timeout handler was delayed by -2.7415977699999985s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18660000000000002s for a response
W20251023 02:05:15.049293 22076 connection.cc:232] RPC call timeout handler was delayed by -2.7064977699999417s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1905s for a response
W20251023 02:05:15.049705 22076 connection.cc:232] RPC call timeout handler was delayed by -2.6209977700000309s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2s for a response
W20251023 02:05:15.050073 22076 connection.cc:232] RPC call timeout handler was delayed by -2.5705977699999494s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2056s for a response
W20251023 02:05:15.050428 22076 connection.cc:232] RPC call timeout handler was delayed by -2.4436977699999716s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2197s for a response
W20251023 02:05:15.050801 22076 connection.cc:232] RPC call timeout handler was delayed by -2.4292977699999483s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
W20251023 02:05:15.051165 22076 connection.cc:232] RPC call timeout handler was delayed by -2.3356977700000243s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23170000000000002s for a response
W20251023 02:05:15.051527 22076 connection.cc:232] RPC call timeout handler was delayed by -2.2969977699999617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.236s for a response
I20251023 02:05:15.010605 24003 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:15.064546 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.064857 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:15.068755 21961 raft_consensus.cc:2393] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:15.070123 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.070470 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.071534 21955 raft_consensus.cc:2393] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:15.065522 21958 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:15.037604 24193 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:15.075246 22028 raft_consensus.cc:2393] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:15.079912 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.033910 21956 raft_consensus.cc:2955] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:15.081012 22037 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:15.083704 22037 raft_consensus.cc:606] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:15.084141 22037 raft_consensus.cc:614] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:15.087467 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.070209 24168 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:15.090443 24168 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:15.083930 24284 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.207s user 0.000s sys 0.001s
W20251023 02:05:15.091765 24279 log.cc:927] Time spent T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20251023 02:05:15.092092 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.092730 21958 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 2.
I20251023 02:05:15.093967 21961 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:15.097307 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.098572 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.074151 24189 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:15.106042 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 114886us
I20251023 02:05:15.080971 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.106464 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.096127 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.099210 22110 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:15.080869 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.095898 22105 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
W20251023 02:05:15.107329 23407 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.002s sys 0.061s
W20251023 02:05:15.108258 23407 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.565s user 0.002s sys 0.061s
I20251023 02:05:15.108799 21957 raft_consensus.cc:2393] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 3: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:15.109666 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.111291 21953 raft_consensus.cc:2393] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:15.111706 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:15.113819 21928 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:15.116991 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.117797 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:15.118718 22104 raft_consensus.cc:2393] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:15.081085 24278 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:15.121093 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.096324 24144 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:15.127457 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.127759 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:15.125434 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.125944 21958 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:15.126264 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.128305 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:15.127490 21955 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:14.935835 22623 rpcz_store.cc:276] 1023 02:05:13.633046 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:13.652338 (+ 19292us) service_pool.cc:225] Handling call
1023 02:05:13.652839 (+ 501us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:13.807503 (+154664us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:13.811664 (+ 4161us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:13.840103 (+ 28439us) log.cc:844] Serialized 26 byte log entry
1023 02:05:14.598671 (+758568us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:14.605022 (+ 6351us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:14.609607 (+ 4585us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:14.897342 (+287735us) raft_consensus.cc:1685] finished
1023 02:05:14.897359 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:14.897502 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":715179,"threads_started":1,"wal-append.queue_time_us":1018937}
I20251023 02:05:15.131124 22108 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:15.133903 22110 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
W20251023 02:05:15.110905 23999 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.019s sys 0.026s
I20251023 02:05:15.129537 22105 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:15.134980 21961 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:15.153280 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.154709 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.156124 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.134076 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.158329 22104 raft_consensus.cc:2393] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
W20251023 02:05:15.141567 22103 consensus_meta.cc:287] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.524s user 0.007s sys 0.000s
I20251023 02:05:15.161151 22103 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:15.162494 22103 raft_consensus.cc:3055] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:15.162873 22103 raft_consensus.cc:740] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:15.163764 22103 consensus_queue.cc:260] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:15.164934 22103 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:15.157362 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.178180 21961 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:15.179481 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:15.180377 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.143929 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.181782 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:15.180542 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 73831.6us
I20251023 02:05:15.173381 22105 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:15.191357 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.194767 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.195770 22110 raft_consensus.cc:2410] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251023 02:05:15.190178 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:15.196935 21957 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:15.197922 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.200084 22108 raft_consensus.cc:2410] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20251023 02:05:15.204439 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:15.144776 23999 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s user 0.019s sys 0.026s
I20251023 02:05:15.205755 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.206425 21961 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:15.188540 22103 raft_consensus.cc:1275] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:05:15.207350 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.211709 22028 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:15.211884 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.212353 22028 raft_consensus.cc:606] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:15.212563 21953 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:15.212755 22028 raft_consensus.cc:614] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251023 02:05:15.198616 24204 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.016s sys 0.061s
I20251023 02:05:15.195535 22616 raft_consensus.cc:1240] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:15.196261 24069 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:15.215289 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.215498 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.225843 21955 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:15.213562 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.229598 21954 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:15.230701 22028 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:15.231115 21928 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:15.231328 22028 raft_consensus.cc:606] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:15.231680 22028 raft_consensus.cc:614] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:15.233062 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:15.180508 23774 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:05:15.241463 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 208800us
I20251023 02:05:15.251614 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:15.213773 24204 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.663s user 0.016s sys 0.061s
I20251023 02:05:15.279340 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.221176 23717 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:15.281787 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:15.282518 21959 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:15.283056 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:15.284098 22111 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.013s sys 0.020s
W20251023 02:05:15.284597 22111 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s user 0.013s sys 0.021s
I20251023 02:05:15.288036 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.289719 21954 raft_consensus.cc:2393] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 5: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:15.281260 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20251023 02:05:15.292831 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 186740us
I20251023 02:05:15.283100 21908 placement_policy_util.cc:244] tablet 5e923b66ccf04dd8985bbfe31adbf0b4: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
I20251023 02:05:15.292233 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:05:15.293807 21955 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:15.278849 24144 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251023 02:05:15.303699 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.306022 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.307345 21953 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
W20251023 02:05:15.297773 24281 log.cc:927] Time spent T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.352s user 0.001s sys 0.001s
W20251023 02:05:15.312489 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 81501.2us
I20251023 02:05:15.311724 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.316097 22037 raft_consensus.cc:2393] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:15.316685 23407 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:15.317303 23407 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:15.319460 23407 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:15.317780 24295 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:15.331759 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:15.281451 23717 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:15.322423 22029 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:15.323045 22028 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:15.334640 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:15.324728 24213 consensus_meta.cc:287] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.902s user 0.008s sys 0.003s
I20251023 02:05:15.326092 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.326578 21908 placement_policy_util.cc:244] tablet 6dddaeefc3d449a0ba9b3e6695e37507: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
I20251023 02:05:15.327414 22103 raft_consensus.cc:1240] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:15.327642 21928 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:15.328012 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:15.359159 22108 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:15.329830 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:05:15.334424 21959 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:15.360245 21954 raft_consensus.cc:1240] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251023 02:05:15.360646 21959 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:15.362123 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:15.335161 23407 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:15.344388 22617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 336) took 1431 ms. Trace:
W20251023 02:05:15.364969 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 73468.1us
W20251023 02:05:15.346246 24186 consensus_meta.cc:287] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.787s user 0.004s sys 0.008s
I20251023 02:05:15.346946 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:15.351876 24217 raft_consensus.cc:1064] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:05:15.377988 22102 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:15.354043 21960 raft_consensus.cc:1240] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:15.367045 23407 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:15.336164 24285 log.cc:927] Time spent T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.275s user 0.000s sys 0.001s
I20251023 02:05:15.383746 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:15.384565 22103 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:15.371346 22104 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:15.363834 21957 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
W20251023 02:05:15.380676 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 44782.6us
I20251023 02:05:15.382476 22589 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:15.386631 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.388310 22104 raft_consensus.cc:2393] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 7: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:15.388620 22589 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
W20251023 02:05:15.389950 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 62399.6us
W20251023 02:05:15.398348 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:15.377986 24069 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:15.433529 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:15.418545 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:15.446879 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:15.515534 21957 raft_consensus.cc:2393] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:15.380369 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:15.386360 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.461825 22077 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:15.424312 24217 raft_consensus.cc:1068] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Unable to promote non-voter 1ab8b0b58147446d84d901b4ed2ff39e: Illegal state: Leader has not yet committed an operation in its own term
I20251023 02:05:15.378463 24186 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:15.443800 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.472923 22103 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:15.428588 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.470804 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.485903 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.535732 22032 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:15.535971 22103 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251023 02:05:15.486203 21959 raft_consensus.cc:1240] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
W20251023 02:05:15.490937 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 124818us
I20251023 02:05:15.451251 24152 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:15.489429 23407 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:15.458691 22108 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:15.365080 24213 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:15.533557 22102 raft_consensus.cc:2410] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251023 02:05:15.533361 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 143765us
W20251023 02:05:15.539633 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 159172us
W20251023 02:05:15.538174 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 149605us
I20251023 02:05:15.546068 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:15.550241 22622 raft_consensus.cc:2410] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 LEARNER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20251023 02:05:15.558282 21954 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20251023 02:05:15.558842 21954 pending_rounds.cc:85] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 7
I20251023 02:05:15.559132 21954 pending_rounds.cc:107] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20251023 02:05:15.559441 21954 raft_consensus.cc:2889] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:15.490835 22623 raft_consensus.cc:1240] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:15.562387 22032 raft_consensus.cc:1240] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:15.563350 21960 pending_rounds.cc:85] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 2
I20251023 02:05:15.563654 21960 pending_rounds.cc:107] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251023 02:05:15.563879 21960 raft_consensus.cc:2889] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:15.491916 24144 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:15.441880 24168 consensus_queue.cc:1059] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.089s
W20251023 02:05:15.569301 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 73615us
I20251023 02:05:15.570431 21925 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:15.570529 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:15.576902 21959 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:15.480118 24291 consensus_queue.cc:1059] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20251023 02:05:15.580989 22034 raft_consensus.cc:2368] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6.
I20251023 02:05:15.578006 22002 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:15.569056 24289 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:15.580417 22077 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:15.606639 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.577209 24146 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:15.609692 23407 consensus_queue.cc:1059] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251023 02:05:15.610517 22000 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:15.533049 24069 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:15.581254 24305 log.cc:927] Time spent T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.235s user 0.000s sys 0.001s
I20251023 02:05:15.618602 22102 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:15.621845 22000 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:15.632473 21959 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:15.636045 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:15.636758 21954 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:15.621204 23407 consensus_queue.cc:1243] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer ca1362b9d9b34073bf35b6f6c9d1159a log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20251023 02:05:15.643280 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:15.644673 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1409) took 1189 ms. Trace:
I20251023 02:05:15.645037 22034 rpcz_store.cc:276] 1023 02:05:14.435811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:14.436020 (+ 209us) service_pool.cc:225] Handling call
1023 02:05:15.624611 (+1188591us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54002a94d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:15.624792 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2567069440}
I20251023 02:05:15.602381 24275 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251023 02:05:15.651582 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 73696.6us
I20251023 02:05:15.652354 22002 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:15.606113 23996 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 02:05:15.608738 23774 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:15.557648 24152 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:15.656988 22000 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:15.578924 24144 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:15.601043 24289 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:15.667223 22110 raft_consensus.cc:1275] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20251023 02:05:15.690114 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 123560us
W20251023 02:05:15.691025 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 120731us
I20251023 02:05:15.696826 23407 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
W20251023 02:05:15.698499 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 46237.3us
I20251023 02:05:15.707985 24213 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:15.708638 24322 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:15.709127 24322 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:15.658034 24275 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:15.665541 23996 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:15.670039 24239 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:15.732578 21954 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
W20251023 02:05:15.768962 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 187694us
I20251023 02:05:15.770355 23407 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:15.672605 24203 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:15.699334 22002 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:15.695799 24112 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:15.658901 24291 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:15.786851 24331 consensus_queue.cc:1059] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251023 02:05:15.788348 24331 raft_consensus.cc:3055] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:05:15.788774 24331 raft_consensus.cc:740] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:15.778846 23996 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:15.793733 24309 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:15.800161 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:15.800732 22110 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:15.654349 23816 consensus_queue.cc:1059] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
W20251023 02:05:15.769443 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 54280.3us
I20251023 02:05:15.709592 24322 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:15.771811 24300 log.cc:927] Time spent T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.244s user 0.000s sys 0.001s
I20251023 02:05:15.364950 22617 rpcz_store.cc:276] 1023 02:05:13.912843 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:13.913043 (+ 200us) service_pool.cc:225] Handling call
1023 02:05:13.913441 (+ 398us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:13.916050 (+ 2609us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:13.916098 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:13.916730 (+ 632us) log.cc:844] Serialized 26 byte log entry
1023 02:05:14.716582 (+799852us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:14.741183 (+ 24601us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:14.775034 (+ 33851us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:15.344126 (+569092us) raft_consensus.cc:1685] finished
1023 02:05:15.344148 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:15.344315 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2059,"raft.queue_time_us":5963,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":5098,"thread_start_us":782138,"threads_started":1,"wal-append.queue_time_us":974478}
I20251023 02:05:15.769398 23888 raft_consensus.cc:3055] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:15.692939 24289 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:15.780293 23717 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:15.815907 24298 log.cc:927] Time spent T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20251023 02:05:15.708678 24152 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:15.797834 21954 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
W20251023 02:05:15.780033 24304 log.cc:927] Time spent T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.382s user 0.000s sys 0.001s
I20251023 02:05:15.825014 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.801466 23407 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:15.782838 24193 consensus_meta.cc:287] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.699s user 0.010s sys 0.000s
I20251023 02:05:15.826035 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1537) took 1327 ms. Trace:
I20251023 02:05:15.804325 24186 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:15.776309 24204 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:15.807204 21925 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:15.795078 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.804848 24144 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:15.803009 24309 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:15.814841 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:15.820279 24069 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:15.822826 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:15.822913 23996 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:15.823462 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:15.825014 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.778093 24275 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:15.836104 22109 rpcz_store.cc:276] 1023 02:05:14.498819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:14.510914 (+ 12095us) service_pool.cc:225] Handling call
1023 02:05:14.521181 (+ 10267us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:14.599464 (+ 78283us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:14.599520 (+ 56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:14.614602 (+ 15082us) log.cc:844] Serialized 26 byte log entry
1023 02:05:15.135965 (+521363us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:15.136016 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:15.136104 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:15.825830 (+689726us) raft_consensus.cc:1685] finished
1023 02:05:15.825861 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:15.825978 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12080,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"thread_start_us":433863,"threads_started":1,"wal-append.queue_time_us":673407}
I20251023 02:05:15.841826 24193 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:15.842307 24189 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:15.842552 24294 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
W20251023 02:05:15.841686 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 409528us
W20251023 02:05:15.882014 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 80232.9us
I20251023 02:05:15.874258 21954 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:15.883931 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:15.884119 21925 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:15.881732 21928 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:15.843721 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 144474us
I20251023 02:05:15.882834 21958 raft_consensus.cc:2393] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 6: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:15.886193 24189 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:15.888227 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:15.888398 24189 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:15.882457 24193 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:15.885424 21954 raft_consensus.cc:1275] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251023 02:05:15.893528 22002 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:15.904611 21954 raft_consensus.cc:1275] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251023 02:05:15.905398 24193 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:15.907371 24193 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:15.908201 24193 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:15.810047 23888 raft_consensus.cc:740] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:15.803165 24146 raft_consensus.cc:697] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:15.904117 21925 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:15.871861 22102 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:15.909416 21928 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:15.932375 24203 raft_consensus.cc:697] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:15.933595 24203 consensus_queue.cc:237] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:15.841919 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 46866.1us
I20251023 02:05:15.935979 22621 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:15.937493 22075 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:15.945783 21961 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.007s sys 0.015s
W20251023 02:05:15.946247 21961 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.584s user 0.007s sys 0.015s
I20251023 02:05:15.826104 24325 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:05:15.947540 24306 log.cc:927] Time spent T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20251023 02:05:15.916034 24294 raft_consensus.cc:697] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:15.949653 23888 consensus_queue.cc:260] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:15.962268 24292 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:15.962796 24292 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:15.962983 24326 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:15.963218 24292 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:15.963452 24326 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:15.963826 24326 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:15.964038 24296 log.cc:927] Time spent T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.430s user 0.000s sys 0.001s
I20251023 02:05:15.966537 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1531) took 1469 ms. Trace:
I20251023 02:05:15.966987 22106 rpcz_store.cc:276] 1023 02:05:14.496922 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:14.497090 (+ 168us) service_pool.cc:225] Handling call
1023 02:05:14.497465 (+ 375us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:14.497754 (+ 289us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:14.497781 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:14.498322 (+ 541us) log.cc:844] Serialized 26 byte log entry
1023 02:05:14.950521 (+452199us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:14.950574 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:14.950647 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:15.966296 (+1015649us) raft_consensus.cc:1685] finished
1023 02:05:15.966321 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:15.966474 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":451803,"threads_started":1,"wal-append.queue_time_us":890930}
W20251023 02:05:15.968737 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 166194us
I20251023 02:05:15.970643 21954 raft_consensus.cc:1240] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:15.873694 24309 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:15.982909 22109 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:15.983027 23407 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:15.983382 23407 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:15.983701 23407 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:15.983724 22104 raft_consensus.cc:1240] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:15.953125 24189 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:15.994040 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:15.994496 22077 leader_election.cc:400] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5.
I20251023 02:05:15.998982 22077 leader_election.cc:403] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:05:15.996138 22029 raft_consensus.cc:1240] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
W20251023 02:05:15.969628 24298 log.cc:927] Time spent T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20251023 02:05:16.004527 22034 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:15.965845 24144 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251023 02:05:16.014668 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:16.009982 22110 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:16.018061 22104 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:15.987422 23888 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:16.027951 24275 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.030843 24189 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:16.031934 24330 consensus_queue.cc:1059] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251023 02:05:16.034036 24329 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:16.034567 24329 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:16.035005 24329 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:16.042192 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:16.057288 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 116175us
W20251023 02:05:16.095872 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 204740us
I20251023 02:05:16.096918 22002 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:15.076280 24003 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:15.180563 24278 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.033303 24330 consensus_queue.cc:1059] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20251023 02:05:15.947706 24325 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:15.786784 24291 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.035972 24188 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:15.356288 24295 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.129814 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:16.143733 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 46914.5us
I20251023 02:05:16.094955 23717 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5.
I20251023 02:05:15.936831 24204 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.159660 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:16.159816 22002 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.165675 22109 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:16.103034 24189 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:16.162583 22077 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:15.949180 24294 consensus_queue.cc:237] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.115401 24186 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:16.167151 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:16.180290 22110 raft_consensus.cc:1275] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20251023 02:05:16.175210 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 272037us
I20251023 02:05:16.175854 22108 raft_consensus.cc:1275] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:05:15.873414 24331 consensus_queue.cc:260] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.053105 24193 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.182408 22031 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
W20251023 02:05:16.171275 24168 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.007s sys 0.043s
I20251023 02:05:16.132517 24112 raft_consensus.cc:697] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:16.188117 21925 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:16.184587 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 169714us
I20251023 02:05:16.189213 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:16.190192 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:16.190601 22110 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:16.184882 24303 log.cc:927] Time spent T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.469s user 0.000s sys 0.001s
I20251023 02:05:16.204231 21957 raft_consensus.cc:1240] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
W20251023 02:05:16.187762 24168 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.007s sys 0.043s
I20251023 02:05:16.183492 24275 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:16.274112 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 59055.2us
I20251023 02:05:16.278501 22002 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.245090 24203 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:16.280017 24203 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:16.279805 24112 consensus_queue.cc:237] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.280447 24203 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:16.245290 23717 consensus_queue.cc:1059] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.038s
W20251023 02:05:16.283617 22111 consensus_meta.cc:287] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.726s user 0.005s sys 0.004s
I20251023 02:05:16.283972 22111 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:16.285001 24346 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251023 02:05:16.285707 24332 log.cc:927] Time spent T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.218s user 0.001s sys 0.000s
I20251023 02:05:16.285776 24324 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251023 02:05:16.286301 24321 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.159s user 0.000s sys 0.000s
I20251023 02:05:16.286816 24324 raft_consensus.cc:3055] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:16.287248 24324 raft_consensus.cc:740] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:16.287405 24204 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:16.118779 24146 consensus_queue.cc:237] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.172536 24188 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.184141 24189 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.288237 24324 consensus_queue.cc:260] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:16.296833 22107 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.016s sys 0.037s
W20251023 02:05:16.318713 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 151584us
W20251023 02:05:16.316605 22107 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.017s sys 0.037s
I20251023 02:05:16.273413 24144 raft_consensus.cc:2804] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251023 02:05:16.325384 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 138339us
I20251023 02:05:16.187395 24186 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:16.325878 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 122354us
I20251023 02:05:16.336478 22077 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:16.344347 22002 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.195417 24335 consensus_queue.cc:1059] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.144s
I20251023 02:05:16.299671 23996 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:16.353257 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:16.326663 24204 raft_consensus.cc:2749] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:16.368639 24003 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:16.368990 24347 consensus_queue.cc:1059] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.163s
I20251023 02:05:16.332489 24144 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:16.371085 23999 raft_consensus.cc:993] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0: : Instructing follower ca1362b9d9b34073bf35b6f6c9d1159a to start an election
I20251023 02:05:16.371426 24349 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:16.371608 24336 log.cc:927] Time spent T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.074s user 0.001s sys 0.001s
W20251023 02:05:16.371989 24328 log.cc:927] Time spent T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.418s user 0.001s sys 0.000s
W20251023 02:05:16.372932 24340 log.cc:927] Time spent T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.267s user 0.001s sys 0.000s
W20251023 02:05:16.373842 24342 log.cc:927] Time spent T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.201s user 0.001s sys 0.000s
I20251023 02:05:16.374009 24330 raft_consensus.cc:3055] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:05:16.370721 22076 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:16.392849 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:16.452518 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:16.404510 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:16.382944 22002 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:16.426308 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 58107.6us
I20251023 02:05:16.428077 22110 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
W20251023 02:05:16.348567 24319 log.cc:927] Time spent T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.437s user 0.001s sys 0.000s
W20251023 02:05:16.433286 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 100007us
I20251023 02:05:16.453832 22077 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:16.494338 21955 raft_consensus.cc:1240] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:16.456187 24168 raft_consensus.cc:2764] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20251023 02:05:16.456768 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 86121.9us
W20251023 02:05:16.498976 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 302548us
I20251023 02:05:16.499007 22076 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:16.458254 24353 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251023 02:05:16.460952 24308 log.cc:927] Time spent T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.761s user 0.002s sys 0.000s
I20251023 02:05:16.467126 24324 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:16.472414 24003 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:16.374465 24338 raft_consensus.cc:3055] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Stepping down as leader of term 2
I20251023 02:05:16.452039 24330 raft_consensus.cc:740] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:16.470752 24203 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.476028 24325 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:16.477095 24295 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:16.477669 23999 raft_consensus.cc:1081] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Signalling peer ca1362b9d9b34073bf35b6f6c9d1159a to start an election
I20251023 02:05:16.477978 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1494) took 1179 ms. Trace:
W20251023 02:05:16.479517 24327 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.686s user 0.010s sys 0.000s
I20251023 02:05:16.481249 24111 raft_consensus.cc:3055] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
W20251023 02:05:16.530973 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 148116us
W20251023 02:05:16.533473 21961 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.001s sys 0.005s
I20251023 02:05:16.505718 24353 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:16.502175 21926 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.027s sys 0.038s
I20251023 02:05:16.358168 24093 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:16.504760 21956 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.012s sys 0.020s
I20251023 02:05:16.371404 24350 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:16.508746 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1489) took 1236 ms. Trace:
I20251023 02:05:16.349767 24152 consensus_queue.cc:1048] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.104s
W20251023 02:05:16.518756 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:16.519752 24338 raft_consensus.cc:740] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:16.521070 21925 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:16.433257 24144 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:16.475104 24348 raft_consensus.cc:1064] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:05:16.434541 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
W20251023 02:05:16.532732 21928 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.000s sys 0.004s
I20251023 02:05:16.532289 24111 raft_consensus.cc:740] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:16.559856 21928 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s user 0.000s sys 0.005s
W20251023 02:05:16.560472 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 651704us
I20251023 02:05:16.527504 21960 rpcz_store.cc:276] 1023 02:05:15.298422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.562828 (+264406us) service_pool.cc:225] Handling call
1023 02:05:15.563158 (+ 330us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:15.571674 (+ 8516us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:15.571702 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:15.574421 (+ 2719us) log.cc:844] Serialized 52 byte log entry
1023 02:05:15.680134 (+105713us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:15.713805 (+ 33671us) log.cc:844] Serialized 14 byte log entry
1023 02:05:15.714007 (+ 202us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:15.714077 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:16.477698 (+763621us) raft_consensus.cc:1685] finished
1023 02:05:16.477725 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:16.477904 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21176,"prepare.queue_time_us":34944,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":40,"raft.queue_time_us":5722,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":58510,"thread_start_us":53952,"threads_started":1,"wal-append.queue_time_us":432619}
I20251023 02:05:16.504523 24359 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251023 02:05:16.542251 21956 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.567s user 0.012s sys 0.023s
I20251023 02:05:16.542446 21958 rpcz_store.cc:276] 1023 02:05:15.197954 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.215054 (+ 17100us) service_pool.cc:225] Handling call
1023 02:05:16.434050 (+1218996us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b540017d754. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:16.434359 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1512251136}
I20251023 02:05:16.536127 24353 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:16.536638 24193 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:16.535241 21926 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.535s user 0.027s sys 0.038s
W20251023 02:05:16.533805 21961 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s user 0.001s sys 0.005s
I20251023 02:05:16.532392 24204 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:16.578500 21928 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.559329 22000 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.525332 24278 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:16.571408 22621 raft_consensus.cc:1240] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 3. Current term is 4. Ops: []
W20251023 02:05:16.575587 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:16.571579 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:16.573463 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 573754us
I20251023 02:05:16.542245 24350 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:16.526089 24003 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:16.577890 24348 raft_consensus.cc:1068] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Unable to promote non-voter 1ab8b0b58147446d84d901b4ed2ff39e: Illegal state: Leader has not yet committed an operation in its own term
I20251023 02:05:16.527019 24360 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
W20251023 02:05:16.585362 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 85736.7us
I20251023 02:05:16.588778 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:16.580740 23407 consensus_meta.cc:287] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.596s user 0.010s sys 0.000s
W20251023 02:05:16.599815 24352 log.cc:927] Time spent T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20251023 02:05:16.583877 24291 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:16.603046 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1426) took 1279 ms. Trace:
I20251023 02:05:16.567987 24359 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:05:16.624591 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 46187.4us
I20251023 02:05:16.606788 23407 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.581348 24186 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:16.608242 24203 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:16.608629 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1154) took 1505 ms. Trace:
I20251023 02:05:16.609076 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1132) took 1803 ms. Trace:
W20251023 02:05:16.611735 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 56382.7us
W20251023 02:05:16.613505 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 54297.5us
I20251023 02:05:16.627139 22624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 341) took 1246 ms. Trace:
I20251023 02:05:16.582952 24357 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:16.588944 24350 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:16.616743 24346 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:16.617733 24303 log.cc:927] Time spent T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.344s user 0.008s sys 0.007s
I20251023 02:05:16.620997 24168 consensus_queue.cc:1059] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20251023 02:05:16.621902 24188 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:16.683768 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:16.684461 21928 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:16.629418 24111 consensus_queue.cc:260] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.686214 24111 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:16.686332 24003 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:16.688901 24329 consensus_meta.cc:287] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.653s user 0.000s sys 0.010s
W20251023 02:05:16.689860 24326 consensus_meta.cc:287] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.726s user 0.009s sys 0.000s
I20251023 02:05:16.690765 24168 consensus_queue.cc:1243] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer 1ab8b0b58147446d84d901b4ed2ff39e log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20251023 02:05:16.567945 24330 consensus_queue.cc:260] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.691426 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251023 02:05:16.693249 24322 consensus_meta.cc:287] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.875s user 0.006s sys 0.001s
I20251023 02:05:16.627879 24359 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:16.695405 23888 consensus_meta.cc:287] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.670s user 0.009s sys 0.001s
W20251023 02:05:16.696476 24343 log.cc:927] Time spent T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20251023 02:05:16.606175 22002 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:16.593982 24339 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:16.701963 24339 raft_consensus.cc:697] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:16.702534 22623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 344) took 1080 ms. Trace:
I20251023 02:05:16.702862 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1435) took 1361 ms. Trace:
I20251023 02:05:16.702930 24339 consensus_queue.cc:237] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.703346 22108 rpcz_store.cc:276] 1023 02:05:15.341297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.537984 (+196687us) service_pool.cc:225] Handling call
1023 02:05:15.538257 (+ 273us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:16.181642 (+643385us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:16.702648 (+521006us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b540025c1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:16.702806 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1446876032}
W20251023 02:05:16.703915 24367 log.cc:927] Time spent T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20251023 02:05:16.612691 22105 rpcz_store.cc:276] 1023 02:05:15.323249 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.332265 (+ 9016us) service_pool.cc:225] Handling call
1023 02:05:15.365744 (+ 33479us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:15.515695 (+149951us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:15.515724 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:15.562074 (+ 46350us) log.cc:844] Serialized 52 byte log entry
1023 02:05:15.772305 (+210231us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b48000b2758. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c649a7971 00007f9c6499ea37 00007f9c6499e8ca 00007f9c64a95a0f 00007f9c64a956d7 00007f9c64aa9cfc 00007f9c64aa9ca8 00007f9c64aa9c28 00007f9c64aa9be3 00007f9c64aa8e8c 00007f9c639e5326
1023 02:05:15.880658 (+108353us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:15.883564 (+ 2906us) log.cc:844] Serialized 14 byte log entry
1023 02:05:15.883757 (+ 193us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:15.883821 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:16.602764 (+718943us) raft_consensus.cc:1685] finished
1023 02:05:16.602850 (+ 86us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:16.602976 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14485,"prepare.queue_time_us":2461,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":76043,"raft.run_cpu_time_us":4054,"raft.run_wall_time_us":321013,"spinlock_wait_cycles":24554240,"thread_start_us":397757,"threads_started":2,"wal-append.queue_time_us":401326}
I20251023 02:05:16.627372 24204 consensus_queue.cc:1059] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.079s
I20251023 02:05:16.630213 22624 rpcz_store.cc:276] 1023 02:05:15.380709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.383026 (+ 2317us) service_pool.cc:225] Handling call
1023 02:05:15.383366 (+ 340us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:16.108988 (+725622us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:16.109019 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:16.109059 (+ 40us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:16.109096 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:16.626935 (+517839us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b5400485b54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:16.626981 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:16.627087 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1668420352}
I20251023 02:05:16.631706 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1496) took 1315 ms. Trace:
I20251023 02:05:16.627713 21925 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.614643 22109 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:16.609753 24189 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:16.635810 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:16.629431 21962 rpcz_store.cc:276] 1023 02:05:14.807084 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:14.817808 (+ 10724us) service_pool.cc:225] Handling call
1023 02:05:14.844389 (+ 26581us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:14.878318 (+ 33929us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:14.878353 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:14.891714 (+ 13361us) log.cc:844] Serialized 52 byte log entry
1023 02:05:15.301408 (+409694us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:15.383638 (+ 82230us) log.cc:844] Serialized 14 byte log entry
1023 02:05:15.581680 (+198042us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:15.581825 (+ 145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:16.460244 (+878419us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:16.607706 (+147462us) spinlock_profiling.cc:243] Waited 3.92 ms on lock 0x7b24000ed0f8. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c83ffe4 00007f9c5c83e0eb 00007f9c5c8cce49 00007f9c5c8ccad5 00007f9c5c8cc030 00007f9c5f65d498 00007f9c5f65dbf8 00007f9c5f635438 00007f9c5f63081a 00007f9c5f62fc66
1023 02:05:16.608874 (+ 1168us) raft_consensus.cc:1685] finished
1023 02:05:16.608894 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:16.609025 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":177668,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":157689,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":17405,"spinlock_wait_cycles":13425664,"thread_start_us":611593,"threads_started":3,"wal-append.queue_time_us":698351}
I20251023 02:05:16.635514 23407 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:16.627527 22077 connection.cc:232] RPC call timeout handler was delayed by -1.4116336949999777s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1703s for a response
W20251023 02:05:16.709425 24309 consensus_meta.cc:287] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.693s user 0.000s sys 0.007s
W20251023 02:05:16.683302 24356 log.cc:927] Time spent T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20251023 02:05:16.698611 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 92540.1us
I20251023 02:05:16.629566 21961 rpcz_store.cc:276] 1023 02:05:15.102935 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.265190 (+162255us) service_pool.cc:225] Handling call
1023 02:05:15.265594 (+ 404us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:15.286875 (+ 21281us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:05:15.946654 (+659779us) log.cc:844] Serialized 14 byte log entry
1023 02:05:16.608261 (+661607us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:16.608310 (+ 49us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:16.608362 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:16.608430 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:16.608553 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31169,"prepare.queue_time_us":186831,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":34789,"raft.run_cpu_time_us":56,"raft.run_wall_time_us":56,"thread_start_us":1202809,"threads_started":2,"wal-append.queue_time_us":644698}
I20251023 02:05:16.626518 24371 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:16.714702 21953 rpcz_store.cc:276] 1023 02:05:15.321679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.641252 (+319573us) service_pool.cc:225] Handling call
1023 02:05:15.667435 (+ 26183us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:15.692031 (+ 24596us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:15.692078 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:15.709146 (+ 17068us) log.cc:844] Serialized 52 byte log entry
1023 02:05:15.970432 (+261286us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:15.996039 (+ 25607us) log.cc:844] Serialized 14 byte log entry
1023 02:05:15.996614 (+ 575us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:15.996756 (+ 142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:16.602858 (+606102us) spinlock_profiling.cc:243] Waited 7.98 ms on lock 0x7b24000ed0f8. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c83ffe4 00007f9c5c83e0eb 00007f9c5c8cce49 00007f9c5c8ccad5 00007f9c5c8cc030 00007f9c5f65d498 00007f9c5f65dbf8 00007f9c5f635438 00007f9c5f63081a 00007f9c5f62fc66
1023 02:05:16.604562 (+ 1704us) raft_consensus.cc:1685] finished
1023 02:05:16.631504 (+ 26942us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:16.631653 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25283,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":325662,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":25756,"spinlock_wait_cycles":18343680,"thread_start_us":260624,"threads_started":1,"wal-append.queue_time_us":327767}
I20251023 02:05:16.716207 24337 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:16.697677 24289 consensus_meta.cc:287] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.883s user 0.010s sys 0.000s
I20251023 02:05:16.721100 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:16.690249 24326 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.723902 24375 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:16.724200 24337 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.690016 24144 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.613031 24338 consensus_queue.cc:260] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.725880 24323 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 02:05:16.726578 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 98935.6us
I20251023 02:05:16.726811 24349 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:16.728261 24331 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:16.729108 24093 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:16.729799 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1158) took 1625 ms. Trace:
I20251023 02:05:16.738731 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1504) took 1052 ms. Trace:
I20251023 02:05:16.719743 24309 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:16.721725 24292 consensus_meta.cc:287] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.758s user 0.001s sys 0.008s
I20251023 02:05:16.765836 24292 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.761654 21956 rpcz_store.cc:276] 1023 02:05:15.686475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.690855 (+ 4380us) service_pool.cc:225] Handling call
1023 02:05:15.707540 (+ 16685us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:15.769712 (+ 62172us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:15.795505 (+ 25793us) log.cc:844] Serialized 14 byte log entry
1023 02:05:16.568811 (+773306us) log.cc:844] Serialized 14 byte log entry
1023 02:05:16.585864 (+ 17053us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:16.585933 (+ 69us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:16.586001 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:16.738453 (+152452us) spinlock_profiling.cc:243] Waited 7.6 ms on lock 0x7b540024b8d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:16.738510 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:16.738665 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47316,"prepare.queue_time_us":33164,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":3265,"raft.queue_time_us":252028,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":6652,"spinlock_wait_cycles":17482624,"thread_start_us":737837,"threads_started":2,"wal-append.queue_time_us":499019}
I20251023 02:05:16.766736 24144 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:16.766896 22106 raft_consensus.cc:1240] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:16.767899 24292 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:16.731774 24317 log.cc:927] Time spent T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 1.079s user 0.007s sys 0.019s
I20251023 02:05:16.769547 24309 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:16.732483 24112 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:16.732853 23407 consensus_queue.cc:1048] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:05:16.771121 24351 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:16.771543 24351 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:16.771906 24351 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:16.736333 22032 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:16.773497 24290 raft_consensus.cc:3055] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:16.773900 24290 raft_consensus.cc:740] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:16.774713 24290 consensus_queue.cc:260] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.754984 24189 consensus_queue.cc:1059] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
W20251023 02:05:16.776733 22031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1172) took 1342 ms (client timeout 1703 ms). Trace:
I20251023 02:05:16.777449 24189 raft_consensus.cc:3055] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:16.777735 24371 raft_consensus.cc:1064] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
I20251023 02:05:16.777869 24189 raft_consensus.cc:740] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:16.778270 24371 raft_consensus.cc:1068] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Unable to promote non-voter 1ab8b0b58147446d84d901b4ed2ff39e: Illegal state: Leader has not yet committed an operation in its own term
I20251023 02:05:16.779000 24189 consensus_queue.cc:260] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.780413 24189 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:16.780949 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 97153.2us
W20251023 02:05:16.777135 22031 rpcz_store.cc:269] 1023 02:05:15.433892 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.434044 (+ 152us) service_pool.cc:225] Handling call
1023 02:05:16.776575 (+1342531us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b540020add4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:16.776677 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1487929088}
I20251023 02:05:16.782096 22000 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:16.782847 23407 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:16.783259 23407 raft_consensus.cc:697] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:16.784008 23407 consensus_queue.cc:237] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.714195 24322 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:16.785148 23816 raft_consensus.cc:3055] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:16.785569 23816 raft_consensus.cc:740] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:16.755550 24375 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.786536 23816 consensus_queue.cc:260] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.787814 23816 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:16.787994 24375 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:16.758967 22002 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.689335 24329 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.790900 24330 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:16.791389 24329 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:16.794622 22106 raft_consensus.cc:1240] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 2. Current term is 3. Ops: []
I20251023 02:05:16.796186 24292 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:16.796876 24322 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:16.796744 24292 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:16.798698 24292 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:16.799186 24292 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:16.799360 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:16.799683 24292 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.802100 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:16.803104 24337 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:16.803194 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:16.779253 24290 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:16.803885 22105 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:16.804754 24335 consensus_queue.cc:1059] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251023 02:05:16.805016 24353 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.805851 24335 consensus_queue.cc:1059] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251023 02:05:16.805986 22587 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.806802 24335 raft_consensus.cc:3055] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:16.807197 24291 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:16.761849 21928 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:16.760946 21955 rpcz_store.cc:276] 1023 02:05:15.104142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.308533 (+204391us) service_pool.cc:225] Handling call
1023 02:05:16.439591 (+1131058us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:16.494915 (+ 55324us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:16.729568 (+234653us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540017d750. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:16.729735 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2526528896}
I20251023 02:05:16.760820 23774 consensus_queue.cc:1048] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20251023 02:05:16.808562 24363 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:16.762573 24368 consensus_queue.cc:1048] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20251023 02:05:16.809049 22587 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:16.764858 24326 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:16.785640 22616 raft_consensus.cc:1240] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:16.810041 24370 raft_consensus.cc:2804] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:16.810433 24370 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:16.810765 24370 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:16.803124 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:16.811810 22106 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:16.762627 21925 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:16.808583 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 46826.7us
W20251023 02:05:16.813896 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 51862.4us
I20251023 02:05:16.814769 24368 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251023 02:05:16.815955 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:16.807161 24335 raft_consensus.cc:740] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:16.817513 23996 consensus_queue.cc:1048] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 02:05:16.824978 22587 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:16.838737 21928 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.838743 22000 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:16.849345 21925 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:16.849964 22000 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:16.850250 21925 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:16.848896 24322 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:16.855010 24322 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:16.855362 24322 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:16.855476 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:16.860167 21928 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.861637 24368 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:16.866649 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:16.867758 22109 raft_consensus.cc:2410] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251023 02:05:16.838601 24353 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:16.868888 21925 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:05:16.869488 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 110640us
I20251023 02:05:16.809322 24351 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.822602 24359 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.878657 21925 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:16.879608 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:16.880157 22109 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:16.870316 21928 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.880826 21925 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:16.852300 24069 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:16.862257 23999 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:16.883292 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:16.862627 24093 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:16.736088 24289 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.886170 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:16.861866 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:16.863265 24368 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:16.868204 24292 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:16.896288 21960 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:16.894085 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:16.899868 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:16.900530 23407 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:16.901506 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:16.901149 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:16.901849 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:16.906113 21953 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 6
I20251023 02:05:16.907902 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:16.878469 24338 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:16.909083 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:16.902822 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:16.909922 22002 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:16.903801 23407 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.915797 21961 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:16.915817 21958 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Advancing to term 3
I20251023 02:05:16.910109 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:16.851753 24295 consensus_queue.cc:1059] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20251023 02:05:16.881757 24144 raft_consensus.cc:2749] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:16.912262 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:16.930974 22103 raft_consensus.cc:2393] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:16.892491 24348 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:16.912864 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:16.912955 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:16.862318 24329 consensus_queue.cc:1059] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
I20251023 02:05:16.914161 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:16.934201 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:16.934671 22105 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:16.913507 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:16.888630 24275 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:16.918311 23407 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:16.918534 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:16.890456 24366 raft_consensus.cc:2804] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:05:16.893043 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:16.893118 24193 raft_consensus.cc:2804] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:05:16.923187 22029 raft_consensus.cc:2368] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8.
I20251023 02:05:16.923432 22035 raft_consensus.cc:1240] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20251023 02:05:16.923497 22617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 343) took 1027 ms. Trace:
I20251023 02:05:16.923954 24294 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:16.924362 24093 raft_consensus.cc:697] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:16.924752 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:16.894189 24362 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20251023 02:05:16.938182 24052 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:16.929150 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:16.929404 22032 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:17.013545 21955 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:16.889806 22108 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd"
dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:39662
I20251023 02:05:16.932126 22034 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:16.895841 24346 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:16.936087 22111 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20251023 02:05:16.936941 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:16.899344 24324 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:05:16.981990 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:16.990127 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:16.990585 21954 raft_consensus.cc:1275] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251023 02:05:16.948419 24239 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:16.913479 23999 raft_consensus.cc:697] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:16.702962 22623 rpcz_store.cc:276] 1023 02:05:15.622093 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.635066 (+ 12973us) service_pool.cc:225] Handling call
1023 02:05:15.709213 (+ 74147us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:16.201270 (+492057us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:16.245091 (+ 43821us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:16.272137 (+ 27046us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:16.279445 (+ 7308us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:16.702251 (+422806us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b5400172854. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:16.702305 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:16.702473 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1094685056}
I20251023 02:05:16.911283 24368 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:16.940019 22030 raft_consensus.cc:1240] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:16.933490 24339 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:16.998020 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:16.888767 24363 consensus_queue.cc:1048] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20251023 02:05:16.927300 22002 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:17.007860 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:17.007992 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:17.009781 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 139553us
I20251023 02:05:16.923974 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:17.002815 24366 raft_consensus.cc:697] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:16.930128 24189 raft_consensus.cc:2749] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:16.950608 24275 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:17.044180 22031 raft_consensus.cc:1240] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:16.948540 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:17.020881 22035 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:17.044884 21962 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:17.026021 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 94380.6us
I20251023 02:05:17.024050 22106 raft_consensus.cc:1240] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:17.046854 22035 raft_consensus.cc:606] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:17.047339 22035 raft_consensus.cc:614] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:17.023370 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1549) took 1191 ms. Trace:
I20251023 02:05:17.048024 22103 rpcz_store.cc:276] 1023 02:05:15.820762 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.823192 (+ 2430us) service_pool.cc:225] Handling call
1023 02:05:17.011559 (+1188367us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54000fcb54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:17.011688 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2430300544}
I20251023 02:05:16.938697 24355 consensus_queue.cc:1048] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.091s
I20251023 02:05:17.005571 22616 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:16.990911 24291 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:17.014575 22108 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 02:05:17.073678 22108 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:17.034350 21928 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:17.034220 22036 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:17.015609 24290 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:16.929102 24350 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.008301 24193 raft_consensus.cc:697] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:17.038527 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 111289us
I20251023 02:05:17.106992 22002 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:17.039597 24364 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20251023 02:05:17.040414 24093 consensus_queue.cc:237] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:16.893810 24335 consensus_queue.cc:260] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:17.048017 24275 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:17.187311 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:17.054493 24188 consensus_queue.cc:1059] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.006s
I20251023 02:05:17.070067 23999 consensus_queue.cc:237] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.074879 23407 raft_consensus.cc:2749] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:17.034171 24239 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.035739 24052 raft_consensus.cc:697] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:17.047672 24339 raft_consensus.cc:697] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:17.012573 24294 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:17.182564 24378 log.cc:927] Time spent T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20251023 02:05:17.214226 24289 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:17.213816 24370 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:17.093021 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:17.218787 21954 raft_consensus.cc:1240] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:17.218650 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:17.222404 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:17.222728 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 115825us
W20251023 02:05:17.236059 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 53259.3us
I20251023 02:05:17.033921 24324 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:05:17.247952 22035 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:17.255296 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:17.255839 22102 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:17.248128 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:17.249053 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:17.256352 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:17.256554 21957 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:17.256918 22109 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:17.250151 22106 raft_consensus.cc:1240] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:17.256575 22107 raft_consensus.cc:2368] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7.
W20251023 02:05:17.258405 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:17.026803 24346 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.178365 24290 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.183660 24291 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:17.278088 24310 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20251023 02:05:17.277977 22108 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.257171 24292 raft_consensus.cc:3055] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:17.285692 22035 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:17.294056 22035 raft_consensus.cc:606] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:17.294427 22109 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:17.287918 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:17.295439 22034 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:17.294435 22035 raft_consensus.cc:614] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:17.269367 24359 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:17.303215 22108 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:17.272265 24351 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:17.318583 24326 consensus_queue.cc:1059] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.008s
I20251023 02:05:17.293090 24292 raft_consensus.cc:740] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:17.339640 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:17.316156 24364 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20251023 02:05:17.374341 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:17.375342 22035 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:17.002561 22617 rpcz_store.cc:276] 1023 02:05:15.896499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:15.896697 (+ 198us) service_pool.cc:225] Handling call
1023 02:05:16.484780 (+588083us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b5800214598. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:05:16.485034 (+ 254us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:16.485311 (+ 277us) raft_consensus.cc:1534] Early marking committed up to index 7
1023 02:05:16.607455 (+122144us) log.cc:844] Serialized 14 byte log entry
1023 02:05:16.877423 (+269968us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:16.888348 (+ 10925us) raft_consensus.cc:1645] Marking committed up to 7
1023 02:05:16.892693 (+ 4345us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:16.910860 (+ 18167us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:16.923442 (+ 12582us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":275689,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":1882,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":10906,"spinlock_wait_cycles":38530432,"thread_start_us":340218,"threads_started":2,"wal-append.queue_time_us":395077}
I20251023 02:05:17.424904 24292 consensus_queue.cc:260] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.314374 24193 consensus_queue.cc:237] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:17.337002 24339 consensus_queue.cc:237] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.283322 24366 consensus_queue.cc:237] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:17.311412 21957 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:17.338898 24052 consensus_queue.cc:237] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.421967 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:17.859094 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:17.450114 22034 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:17.868057 22108 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
W20251023 02:05:17.539750 21953 consensus_meta.cc:287] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.624s user 0.009s sys 0.005s
I20251023 02:05:17.864410 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:17.499527 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:17.863533 21956 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.012s sys 0.008s
W20251023 02:05:17.571163 21960 consensus_meta.cc:287] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.668s user 0.003s sys 0.010s
W20251023 02:05:17.574883 22589 leader_election.cc:336] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.792s (SENT)
I20251023 02:05:17.870481 21960 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
W20251023 02:05:17.870347 21956 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s user 0.012s sys 0.008s
W20251023 02:05:17.503479 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 469203us
I20251023 02:05:17.869096 21953 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:17.871707 21928 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:17.870602 22589 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:17.855983 24397 log.cc:927] Time spent T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.377s user 0.000s sys 0.000s
I20251023 02:05:17.426789 24377 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:17.862629 24291 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:17.410028 24003 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:17.424985 24387 log.cc:927] Time spent T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20251023 02:05:17.873793 24188 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:17.884536 24188 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.886651 24188 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:17.888044 22032 raft_consensus.cc:1275] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20251023 02:05:17.888136 21908 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20251023 02:05:17.889159 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:17.889775 22032 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:17.891502 24370 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:17.892678 24335 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:17.874454 24213 raft_consensus.cc:2804] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251023 02:05:17.894450 24349 consensus_meta.cc:287] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.069s user 0.010s sys 0.000s
I20251023 02:05:17.894876 24349 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:17.895090 24213 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:17.895534 24213 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:17.874576 24377 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.904167 24377 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:17.872299 22031 raft_consensus.cc:2393] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:17.874993 22617 raft_consensus.cc:1240] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:17.879392 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:17.907455 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:17.908159 21953 raft_consensus.cc:2393] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:17.908918 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:17.906157 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:17.883234 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:17.909921 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:17.910902 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:17.911146 24069 consensus_meta.cc:287] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.018s user 0.004s sys 0.011s
I20251023 02:05:17.911509 22030 raft_consensus.cc:3055] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:17.912101 22030 raft_consensus.cc:740] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:17.912695 24322 consensus_meta.cc:287] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 1.057s user 0.013s sys 0.005s
I20251023 02:05:17.913075 24322 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:17.909570 22031 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:17.915455 24322 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:17.635452 24347 raft_consensus.cc:3055] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:17.861994 24290 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:17.604990 24239 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:17.917264 24292 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:17.912843 22030 consensus_queue.cc:260] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.863794 24294 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:17.632259 24359 raft_consensus.cc:3055] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:17.882218 24144 raft_consensus.cc:3055] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:17.882161 24324 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:17.919320 24213 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.919814 22587 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:17.813997 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:17.921820 22032 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
W20251023 02:05:17.922449 22102 consensus_meta.cc:287] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.646s user 0.004s sys 0.004s
I20251023 02:05:17.923041 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:17.676242 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 93045.9us
W20251023 02:05:17.971554 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 51838.6us
I20251023 02:05:17.925470 22031 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:17.926075 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:17.972409 22104 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:17.911561 24069 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251023 02:05:17.928934 21955 consensus_meta.cc:287] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.910s user 0.010s sys 0.005s
I20251023 02:05:17.921375 24377 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:17.984062 21955 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:17.986088 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1530) took 1094 ms. Trace:
W20251023 02:05:17.932873 22105 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.010s sys 0.028s
W20251023 02:05:17.932145 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:17.960028 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 385192us
W20251023 02:05:17.987687 22105 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.021s sys 0.043s
W20251023 02:05:17.988562 22589 leader_election.cc:336] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.910s (SENT)
I20251023 02:05:17.989010 22589 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:17.930716 22033 consensus_meta.cc:287] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.894s user 0.010s sys 0.009s
W20251023 02:05:17.930408 21962 consensus_meta.cc:287] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.885s user 0.012s sys 0.008s
I20251023 02:05:17.989741 22033 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:17.989931 21962 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:17.906184 22618 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:17.965022 24347 raft_consensus.cc:740] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:17.963411 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 91370us
I20251023 02:05:17.964473 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1529) took 1076 ms. Trace:
I20251023 02:05:17.996495 21957 rpcz_store.cc:276] 1023 02:05:16.887897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:16.914429 (+ 26532us) service_pool.cc:225] Handling call
1023 02:05:17.964319 (+1049890us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b540030e8d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:17.964420 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":547807488}
I20251023 02:05:17.966240 24144 raft_consensus.cc:740] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:17.885458 22616 consensus_meta.cc:287] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.815s user 0.005s sys 0.003s
W20251023 02:05:17.966091 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:17.984576 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:17.971403 22102 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:17.973034 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:17.979096 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:17.970114 23407 consensus_queue.cc:1059] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251023 02:05:17.911000 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.007097 22623 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
W20251023 02:05:17.924047 24275 consensus_meta.cc:287] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.709s user 0.012s sys 0.008s
I20251023 02:05:17.878752 24003 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:17.980465 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:17.981855 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:18.014006 22032 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:17.986110 21958 consensus_meta.cc:287] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.069s user 0.010s sys 0.009s
I20251023 02:05:18.014577 21958 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:18.015064 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1522) took 1130 ms. Trace:
I20251023 02:05:18.015353 21958 rpcz_store.cc:276] 1023 02:05:16.884456 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:16.886607 (+ 2151us) service_pool.cc:225] Handling call
1023 02:05:18.015004 (+1128397us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.016075 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:17.990687 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1528) took 1103 ms. Trace:
W20251023 02:05:17.962260 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 90645.6us
I20251023 02:05:18.016538 21962 rpcz_store.cc:276] 1023 02:05:16.887681 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:16.927855 (+ 40174us) service_pool.cc:225] Handling call
1023 02:05:17.990613 (+1062758us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:17.992460 22000 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:17.995492 22029 raft_consensus.cc:1240] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:18.017480 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.018680 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.019343 21962 raft_consensus.cc:2393] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:18.019469 24213 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:17.963431 24359 raft_consensus.cc:740] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:17.965690 24335 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:17.962947 22030 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:17.998122 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.054576 21957 raft_consensus.cc:2393] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:05:17.481159 24386 log.cc:927] Time spent T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.295s user 0.000s sys 0.001s
W20251023 02:05:18.000916 22107 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.019s sys 0.034s
I20251023 02:05:17.968230 24290 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:17.967183 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:18.055824 22107 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s user 0.019s sys 0.034s
I20251023 02:05:18.056205 22108 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:17.987025 24322 raft_consensus.cc:2804] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:17.421892 24350 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:18.013942 21954 raft_consensus.cc:2376] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7. Re-sending same reply.
I20251023 02:05:17.982324 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:17.873534 24346 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:18.020025 24323 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:18.049813 22104 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:18.001744 24312 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:17.992945 24377 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:17.986984 21955 rpcz_store.cc:276] 1023 02:05:16.891409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:16.903472 (+ 12063us) service_pool.cc:225] Handling call
1023 02:05:17.985990 (+1082518us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.060572 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.666137 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.666716 21958 raft_consensus.cc:2393] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:18.666944 21962 raft_consensus.cc:2410] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20251023 02:05:18.667222 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1179) took 1411 ms. Trace:
W20251023 02:05:18.358242 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 166031us
I20251023 02:05:18.667703 21958 rpcz_store.cc:276] 1023 02:05:17.255659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.016744 (+761085us) service_pool.cc:225] Handling call
1023 02:05:18.667167 (+650423us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:18.409301 24366 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.508s user 0.000s sys 0.015s
W20251023 02:05:18.409606 24052 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.006s sys 0.018s
W20251023 02:05:18.669430 24052 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.792s user 0.009s sys 0.021s
I20251023 02:05:18.193871 24359 consensus_queue.cc:260] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:18.416929 22615 consensus_meta.cc:287] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 1.200s user 0.009s sys 0.010s
I20251023 02:05:18.670856 24359 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:18.670914 22615 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3.
I20251023 02:05:18.077050 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:18.435348 24193 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.011s sys 0.005s
W20251023 02:05:18.673936 24193 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.791s user 0.018s sys 0.007s
I20251023 02:05:18.673810 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1180) took 1417 ms. Trace:
W20251023 02:05:18.443809 22077 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.011s sys 0.005s
W20251023 02:05:18.675102 22077 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.743s user 0.016s sys 0.009s
W20251023 02:05:18.668581 24406 log.cc:927] Time spent T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.312s user 0.000s sys 0.000s
W20251023 02:05:18.415028 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 292689us
I20251023 02:05:18.521183 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:18.677260 24188 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:18.677449 22076 leader_election.cc:336] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.502s (SENT)
W20251023 02:05:18.677680 21961 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.000s sys 0.024s
W20251023 02:05:18.678040 21961 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.746s user 0.000s sys 0.024s
I20251023 02:05:18.677707 23717 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:18.678463 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 661627us
I20251023 02:05:18.678730 23717 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:18.679723 21928 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:18.580003 24292 consensus_meta.cc:287] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.615s user 0.009s sys 0.001s
I20251023 02:05:18.681030 24292 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:18.003932 22616 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:18.677896 24188 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:18.681982 22616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 240) took 1842 ms (client timeout 1613 ms). Trace:
I20251023 02:05:18.066691 24290 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:18.682520 22616 rpcz_store.cc:269] 1023 02:05:16.839162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:16.849456 (+ 10294us) service_pool.cc:225] Handling call
1023 02:05:18.681879 (+1832423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.683811 24188 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:18.025660 24275 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:18.684242 24290 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:18.681139 24391 raft_consensus.cc:2749] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:18.170805 22108 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:18.681031 23717 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:18.261281 22030 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:18.685952 24275 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:18.449445 22111 raft_consensus.cc:1240] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:18.687240 24275 consensus_queue.cc:1048] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:18.025401 22032 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:18.688898 24275 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:18.127107 24003 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:18.690141 22000 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:18.691617 24405 raft_consensus.cc:2749] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:18.691951 24377 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:18.455026 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 338683us
I20251023 02:05:18.127585 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.693284 24069 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:18.693748 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1167) took 1463 ms. Trace:
W20251023 02:05:18.694422 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:18.694828 24213 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
W20251023 02:05:18.600111 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:18.695446 24213 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:18.696172 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:18.486812 24353 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:18.697556 24353 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:18.699841 24353 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:18.693796 24069 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:18.703584 24069 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:18.692948 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:18.667239 21954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1175) took 1423 ms (client timeout 1777 ms). Trace:
W20251023 02:05:18.705395 21954 rpcz_store.cc:269] 1023 02:05:17.243867 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:17.980140 (+736273us) service_pool.cc:225] Handling call
1023 02:05:18.667183 (+687043us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.066828 24322 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:18.706632 24213 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:18.666596 21960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1176) took 1422 ms (client timeout 1783 ms). Trace:
W20251023 02:05:18.708740 21960 rpcz_store.cc:269] 1023 02:05:17.244139 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:17.982053 (+737914us) service_pool.cc:225] Handling call
1023 02:05:18.666537 (+684484us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:18.348201 24338 consensus_meta.cc:287] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 1.419s user 0.019s sys 0.000s
W20251023 02:05:18.668396 22589 leader_election.cc:336] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 2.095s (SENT)
I20251023 02:05:18.803467 22589 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:18.521606 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:18.804822 22110 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:18.521935 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:18.674515 21953 rpcz_store.cc:276] 1023 02:05:17.255968 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.017003 (+761035us) service_pool.cc:225] Handling call
1023 02:05:18.673740 (+656737us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.805953 22103 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:18.694236 21957 rpcz_store.cc:276] 1023 02:05:17.230818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.059444 (+828626us) service_pool.cc:225] Handling call
1023 02:05:18.693671 (+634227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.677973 22076 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:18.808224 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 130808us
W20251023 02:05:18.808871 22076 leader_election.cc:336] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.521s (SENT)
I20251023 02:05:18.809232 22076 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:18.810154 22076 leader_election.cc:336] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.525s (SENT)
W20251023 02:05:18.810889 22076 leader_election.cc:336] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.606s (SENT)
I20251023 02:05:18.811237 22076 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:18.811906 23717 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:18.681149 21928 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:18.813150 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 132123us
I20251023 02:05:18.718600 22000 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:18.815143 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 96631.6us
I20251023 02:05:18.749738 22077 leader_election.cc:400] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8.
I20251023 02:05:18.817847 22077 leader_election.cc:403] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20251023 02:05:18.753129 22106 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:18.707149 24322 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:18.681612 24292 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:18.804255 21962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1181) took 1545 ms (client timeout 2056 ms). Trace:
W20251023 02:05:18.852921 21962 rpcz_store.cc:269] 1023 02:05:17.259264 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.019783 (+760519us) service_pool.cc:225] Handling call
1023 02:05:18.804182 (+784399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.826632 22587 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:18.425118 24395 log.cc:927] Time spent T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.231s user 0.000s sys 0.000s
I20251023 02:05:18.830340 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1446) took 1603 ms. Trace:
W20251023 02:05:18.564383 24389 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.186s user 0.000s sys 0.022s
W20251023 02:05:18.712733 23999 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.007s sys 0.014s
W20251023 02:05:18.418113 24339 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.535s user 0.007s sys 0.007s
I20251023 02:05:18.872507 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:18.831050 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 162714us
I20251023 02:05:18.873247 22106 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 8.
I20251023 02:05:18.873279 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.817274 22000 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:18.874159 21953 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
W20251023 02:05:18.874738 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 57548.1us
I20251023 02:05:18.834977 21928 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:18.829463 24003 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:18.829629 24193 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20251023 02:05:18.846186 24052 raft_consensus.cc:2749] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:18.849562 24331 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:18.374437 24403 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:18.850080 24290 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251023 02:05:18.662240 24398 log.cc:927] Time spent T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.305s user 0.000s sys 0.001s
I20251023 02:05:18.747565 24370 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251023 02:05:18.850201 24312 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:18.090245 24347 consensus_queue.cc:260] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:18.129868 24144 consensus_queue.cc:260] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:18.872725 24188 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
W20251023 02:05:18.855063 22076 leader_election.cc:336] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.716s (SENT)
I20251023 02:05:18.856679 22105 rpcz_store.cc:276] 1023 02:05:17.226684 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:17.257096 (+ 30412us) service_pool.cc:225] Handling call
1023 02:05:17.283671 (+ 26575us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:17.304477 (+ 20806us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:17.304513 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:17.304931 (+ 418us) log.cc:844] Serialized 26 byte log entry
1023 02:05:17.988269 (+683338us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:17.988327 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:18.805258 (+816931us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b54000fc654. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:18.805322 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:18.805387 (+ 65us) raft_consensus.cc:1685] finished
1023 02:05:18.805408 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:18.830266 (+ 24858us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4156,"raft.queue_time_us":11378,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":2462,"spinlock_wait_cycles":43091840,"thread_start_us":667753,"threads_started":1,"wal-append.queue_time_us":868773}
I20251023 02:05:18.879315 24405 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251023 02:05:18.871423 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 121916us
I20251023 02:05:18.899742 22076 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:18.876219 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.877769 22000 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:18.878002 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.903113 21962 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:18.903769 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1191) took 1023 ms. Trace:
I20251023 02:05:18.901427 22077 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:18.904160 21962 rpcz_store.cc:276] 1023 02:05:17.880518 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.854248 (+973730us) service_pool.cc:225] Handling call
1023 02:05:18.903686 (+ 49438us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.830022 24353 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:18.870945 24389 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.492s user 0.000s sys 0.023s
W20251023 02:05:18.870476 23999 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.952s user 0.007s sys 0.014s
I20251023 02:05:18.853963 24346 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251023 02:05:18.918929 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 223042us
I20251023 02:05:18.906456 24144 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:18.925483 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:18.926880 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:18.927706 22111 raft_consensus.cc:2393] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:18.927783 22077 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:18.928310 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:18.930150 22104 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
W20251023 02:05:18.883832 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 48954.1us
I20251023 02:05:18.931429 22104 raft_consensus.cc:1240] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:18.931566 21928 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:18.885248 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1532) took 1989 ms. Trace:
I20251023 02:05:18.856379 24350 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:18.856700 24323 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:18.889067 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.934821 21953 raft_consensus.cc:2393] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:18.872460 24338 raft_consensus.cc:2804] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader election won for term 5
W20251023 02:05:18.869297 24093 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.011s sys 0.014s
I20251023 02:05:18.899175 22103 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:18.898392 24052 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:18.932199 21956 rpcz_store.cc:276] 1023 02:05:16.895666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:17.033590 (+137924us) service_pool.cc:225] Handling call
1023 02:05:17.046624 (+ 13034us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:17.062745 (+ 16121us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:17.062788 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:17.285140 (+222352us) log.cc:844] Serialized 52 byte log entry
1023 02:05:17.872845 (+587705us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:18.459210 (+586365us) log.cc:844] Serialized 14 byte log entry
1023 02:05:18.459528 (+ 318us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:18.459627 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:18.885026 (+425399us) raft_consensus.cc:1685] finished
1023 02:05:18.885046 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:18.885185 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58629,"prepare.queue_time_us":357106,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":251229,"raft.run_cpu_time_us":555,"raft.run_wall_time_us":110513,"thread_start_us":1036854,"threads_started":2,"wal-append.queue_time_us":904964}
I20251023 02:05:18.933384 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.942703 21928 leader_election.cc:400] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6.
I20251023 02:05:18.943104 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:18.943590 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:18.937297 21925 leader_election.cc:400] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7.
I20251023 02:05:18.902650 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1189) took 1022 ms. Trace:
I20251023 02:05:18.944278 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1195) took 1062 ms. Trace:
I20251023 02:05:18.884038 24331 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:18.909677 24213 raft_consensus.cc:2749] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:18.941816 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:18.904214 24405 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:18.943226 21928 leader_election.cc:403] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:05:18.944458 21957 rpcz_store.cc:276] 1023 02:05:17.879865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.830852 (+950987us) service_pool.cc:225] Handling call
1023 02:05:18.902588 (+ 71736us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.959052 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:18.959954 21957 raft_consensus.cc:2393] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:18.960875 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.938303 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1184) took 1637 ms. Trace:
I20251023 02:05:18.943962 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.961848 21953 rpcz_store.cc:276] 1023 02:05:17.301091 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.879982 (+1578891us) service_pool.cc:225] Handling call
1023 02:05:18.938236 (+ 58254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.962756 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:18.963035 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:18.939981 23717 raft_consensus.cc:2749] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8.
I20251023 02:05:18.944664 21955 rpcz_store.cc:276] 1023 02:05:17.927393 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.943375 (+1015982us) service_pool.cc:225] Handling call
1023 02:05:18.944226 (+ 851us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.964414 24003 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:18.909103 24290 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:18.965196 24290 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:18.964947 24003 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:18.965682 24290 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:18.967118 24003 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:18.942728 24339 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:18.924057 24346 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:18.900972 24393 consensus_queue.cc:1059] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20251023 02:05:18.968588 24346 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:18.969172 24393 raft_consensus.cc:3055] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:18.969520 24393 raft_consensus.cc:740] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:18.969815 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:18.970372 24393 consensus_queue.cc:260] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:18.971664 24393 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:18.947058 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1531) took 2055 ms. Trace:
I20251023 02:05:18.954699 22105 raft_consensus.cc:1240] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20251023 02:05:18.972223 21961 rpcz_store.cc:276] 1023 02:05:16.891712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:17.029822 (+138110us) service_pool.cc:225] Handling call
1023 02:05:17.041959 (+ 12137us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:17.055019 (+ 13060us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:17.055083 (+ 64us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:17.063328 (+ 8245us) log.cc:844] Serialized 52 byte log entry
1023 02:05:17.858469 (+795141us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:17.883123 (+ 24654us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b48000b22d8. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c649a7971 00007f9c6499ea37 00007f9c6499e8ca 00007f9c64a95a0f 00007f9c64a956d7 00007f9c64aa9cfc 00007f9c64aa9ca8 00007f9c64aa9c28 00007f9c64aa9be3 00007f9c64aa8e8c 00007f9c639e5326
1023 02:05:18.678588 (+795465us) log.cc:844] Serialized 14 byte log entry
1023 02:05:18.678786 (+ 198us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:18.678864 (+ 78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:18.941527 (+262663us) raft_consensus.cc:1685] finished
1023 02:05:18.946844 (+ 5317us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:18.946996 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":371277,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":151096,"raft.run_cpu_time_us":10092,"raft.run_wall_time_us":692072,"spinlock_wait_cycles":68799360,"thread_start_us":1008096,"threads_started":2,"wal-append.queue_time_us":984764}
I20251023 02:05:18.973887 24213 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:18.919328 24292 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:18.939678 24350 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:18.974426 24213 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:18.886117 24403 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:18.957118 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.975891 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:18.976325 21962 raft_consensus.cc:2368] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5.
I20251023 02:05:18.976521 22032 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:18.976743 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1185) took 1651 ms. Trace:
I20251023 02:05:18.976997 24403 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:18.977073 21962 rpcz_store.cc:276] 1023 02:05:17.337825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.943714 (+1605889us) service_pool.cc:225] Handling call
1023 02:05:18.976692 (+ 32978us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.977440 21956 raft_consensus.cc:2376] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7. Re-sending same reply.
I20251023 02:05:18.946904 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:18.947395 24323 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:18.978371 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.979420 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.979578 24323 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:18.973373 23999 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:18.980389 23999 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:18.944468 24338 raft_consensus.cc:697] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:18.981671 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:18.982441 23999 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:18.982677 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 228) took 1685 ms. Trace:
I20251023 02:05:18.974293 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.971674 24312 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:18.984057 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.983069 22109 rpcz_store.cc:276] 1023 02:05:17.297455 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:17.330592 (+ 33137us) service_pool.cc:225] Handling call
1023 02:05:17.331955 (+ 1363us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:17.857927 (+525972us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:17.857972 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:17.881763 (+ 23791us) log.cc:844] Serialized 26 byte log entry
1023 02:05:18.019493 (+137730us) spinlock_profiling.cc:243] Waited 51.5 ms on lock 0x7b18004c31e0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9cdd37 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c5f62f2bb 00007f9c5f634dc3 00007f9c5f635171 00007f9c5fa5465f
1023 02:05:18.425694 (+406201us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:18.425739 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:18.425822 (+ 83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:18.982426 (+556604us) raft_consensus.cc:1685] finished
1023 02:05:18.982448 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:18.982628 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30409,"raft.queue_time_us":35639,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":9211,"spinlock_wait_cycles":118526208,"thread_start_us":405950,"threads_started":1,"wal-append.queue_time_us":1099210}
I20251023 02:05:18.983940 22590 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:18.984664 21961 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
W20251023 02:05:18.939354 24093 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.821s user 0.012s sys 0.014s
I20251023 02:05:18.985540 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.986104 22034 raft_consensus.cc:606] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:18.964846 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:18.986572 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:18.986786 21955 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:18.975930 24350 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:18.983130 23999 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:18.988677 23999 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:18.989765 22617 raft_consensus.cc:2410] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20251023 02:05:18.990204 24312 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:18.979933 21962 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:18.991590 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1192) took 1110 ms. Trace:
W20251023 02:05:18.991729 24410 log.cc:927] Time spent T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20251023 02:05:18.991930 21962 rpcz_store.cc:276] 1023 02:05:17.880780 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.979173 (+1098393us) service_pool.cc:225] Handling call
1023 02:05:18.991536 (+ 12363us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.992267 24405 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:18.992823 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.992355 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:18.994621 21961 raft_consensus.cc:2393] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:18.994094 24416 consensus_queue.cc:1059] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:05:18.906898 24188 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:19.000129 24188 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:19.000646 24188 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:18.993325 23999 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:19.004801 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.005352 21961 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:18.945516 21925 leader_election.cc:403] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20251023 02:05:19.006198 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 69025.3us
W20251023 02:05:19.037523 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:19.047127 21955 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:19.047979 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1538) took 1069 ms. Trace:
I20251023 02:05:19.048391 21955 rpcz_store.cc:276] 1023 02:05:17.978614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.964631 (+986017us) service_pool.cc:225] Handling call
1023 02:05:19.047887 (+ 83256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:18.934217 24418 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251023 02:05:19.066720 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 139026us
I20251023 02:05:18.939002 24389 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:19.067070 21925 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:19.071175 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:19.065788 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 210767us
I20251023 02:05:19.073235 21925 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:19.075098 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:19.075825 21957 raft_consensus.cc:2468] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
W20251023 02:05:19.074420 22076 leader_election.cc:336] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.737s (SENT)
W20251023 02:05:19.079308 22076 leader_election.cc:336] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.740s (SENT)
I20251023 02:05:19.079875 22076 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:19.080596 24368 consensus_meta.cc:287] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 2.019s user 0.016s sys 0.005s
W20251023 02:05:19.082885 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 177108us
I20251023 02:05:18.976343 24213 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:19.085405 22109 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
W20251023 02:05:19.085309 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 147465us
W20251023 02:05:19.100667 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 274164us
I20251023 02:05:19.087713 24292 consensus_queue.cc:1059] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20251023 02:05:19.086977 24052 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:19.064600 24193 raft_consensus.cc:2749] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7.
I20251023 02:05:19.067469 24393 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6.
I20251023 02:05:19.092077 22103 raft_consensus.cc:1275] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251023 02:05:19.092085 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.094523 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:19.080809 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:18.939163 24353 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:19.075428 24403 raft_consensus.cc:2804] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:19.115125 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:19.099522 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:19.101903 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:19.104100 22587 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:19.116183 22000 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:19.086506 22077 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:19.106242 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:19.116961 22105 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:19.109555 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.117841 21958 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:19.118872 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:19.119295 22037 raft_consensus.cc:2393] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:19.111363 22002 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:19.112407 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:19.112969 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:19.120271 22000 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:19.085139 24389 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:19.125633 22028 raft_consensus.cc:2393] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:19.069916 24290 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.070617 24323 consensus_queue.cc:1059] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20251023 02:05:18.982479 24338 consensus_queue.cc:237] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:19.102788 24377 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:19.127856 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:19.128028 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 48745.7us
I20251023 02:05:19.128835 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:19.119689 22032 raft_consensus.cc:2393] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:19.106360 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:19.131085 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:19.132186 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:19.132319 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:19.138553 22587 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:19.142409 24323 raft_consensus.cc:3055] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:19.139843 21962 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:19.142753 24323 raft_consensus.cc:740] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:19.143596 24323 consensus_queue.cc:260] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.145220 24323 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:19.146149 24421 log.cc:927] Time spent T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20251023 02:05:19.147704 24292 raft_consensus.cc:3055] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:19.148186 24292 raft_consensus.cc:740] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:19.126988 23996 consensus_queue.cc:1059] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 6.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20251023 02:05:19.115990 24052 raft_consensus.cc:697] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:19.154956 24052 consensus_queue.cc:237] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.125259 24193 consensus_queue.cc:1059] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20251023 02:05:19.158632 24366 raft_consensus.cc:3055] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:19.158980 24366 raft_consensus.cc:740] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:19.118238 24350 consensus_queue.cc:1059] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20251023 02:05:19.159726 24366 consensus_queue.cc:260] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.160562 24350 raft_consensus.cc:3055] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:19.160909 24350 raft_consensus.cc:740] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:19.161715 24350 consensus_queue.cc:260] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:19.163182 24350 raft_consensus.cc:3060] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:19.163626 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1448) took 1929 ms. Trace:
W20251023 02:05:19.165916 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 79490.6us
I20251023 02:05:19.141285 21955 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:19.168838 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.169672 21955 raft_consensus.cc:2376] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 3. Re-sending same reply.
I20251023 02:05:19.170637 21961 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:19.171350 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.163946 22107 rpcz_store.cc:276] 1023 02:05:17.234604 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:17.259021 (+ 24417us) service_pool.cc:225] Handling call
1023 02:05:17.259289 (+ 268us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:17.284423 (+ 25134us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:17.284441 (+ 18us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:17.289642 (+ 5201us) log.cc:844] Serialized 26 byte log entry
1023 02:05:18.056458 (+766816us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:18.056514 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:19.163259 (+1106745us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b54001ce8d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:19.163309 (+ 50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:19.163356 (+ 47us) raft_consensus.cc:1685] finished
1023 02:05:19.163370 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:19.163574 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57335552,"thread_start_us":717180,"threads_started":1,"wal-append.queue_time_us":873599}
I20251023 02:05:19.102711 24368 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.173046 21955 raft_consensus.cc:2376] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5. Re-sending same reply.
I20251023 02:05:19.172906 22002 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:05:19.175345 22076 leader_election.cc:336] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.777s (SENT)
W20251023 02:05:19.176354 22076 leader_election.cc:336] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.783s (SENT)
I20251023 02:05:19.176826 22076 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:19.180473 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.149209 24292 consensus_queue.cc:260] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.191337 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.129509 24437 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:19.191893 21955 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:19.192376 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 89944us
I20251023 02:05:19.196988 22623 raft_consensus.cc:2393] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:19.204603 21955 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:19.186417 21958 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:19.134945 24403 raft_consensus.cc:697] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:19.197639 21953 raft_consensus.cc:2393] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:19.202047 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1186) took 2270 ms. Trace:
I20251023 02:05:19.207180 22029 rpcz_store.cc:276] 1023 02:05:16.928663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:17.004228 (+ 75565us) service_pool.cc:225] Handling call
1023 02:05:17.021483 (+ 17255us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:18.017307 (+995824us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:19.199222 (+1181915us) spinlock_profiling.cc:243] Waited 974 ms on lock 0x7b540034b554. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:19.199369 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2239580416}
I20251023 02:05:19.202486 22037 pending_rounds.cc:85] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: Aborting all ops after (but not including) 4
I20251023 02:05:19.188282 24430 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:19.185873 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.185868 24431 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:19.192975 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:19.209681 21957 raft_consensus.cc:2376] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6. Re-sending same reply.
I20251023 02:05:19.189141 24292 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:19.193044 24425 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:19.208329 22037 pending_rounds.cc:107] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 4.5
W20251023 02:05:19.211742 21958 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1188) took 1332 ms (client timeout 1613 ms). Trace:
I20251023 02:05:19.217809 22037 raft_consensus.cc:2917] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Aborting config change with OpId 4.5: Aborted: Op aborted by new leader
I20251023 02:05:19.218295 22037 raft_consensus.cc:2889] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:19.218713 24402 raft_consensus.cc:3055] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:19.219146 24402 raft_consensus.cc:740] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:19.219841 21958 rpcz_store.cc:269] 1023 02:05:17.879498 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.096322 (+1216824us) service_pool.cc:225] Handling call
1023 02:05:19.211678 (+115356us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:19.213838 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.189622 24416 consensus_queue.cc:1059] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20251023 02:05:19.223302 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:19.224721 24425 raft_consensus.cc:697] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:19.224957 24436 raft_consensus.cc:2749] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:19.224937 24402 consensus_queue.cc:260] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.2, Last appended by leader: 2, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:19.226069 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 49773.6us
I20251023 02:05:19.225857 24425 consensus_queue.cc:237] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.226518 24402 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:19.226799 22076 leader_election.cc:336] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.855s (SENT)
I20251023 02:05:19.227202 22076 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:19.230841 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:19.220873 21961 raft_consensus.cc:2376] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6. Re-sending same reply.
I20251023 02:05:19.215204 22002 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:19.233222 22002 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:19.234553 22002 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:19.236794 22002 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:19.212803 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:19.245693 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:19.247044 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 72156.4us
I20251023 02:05:19.209650 24427 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:19.217119 24424 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:19.215741 24213 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:19.248379 21925 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:19.215291 24431 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:05:19.252208 22076 leader_election.cc:336] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.866s (SENT)
I20251023 02:05:19.260753 22076 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:19.252465 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.261889 21961 raft_consensus.cc:2376] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6. Re-sending same reply.
I20251023 02:05:19.254855 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:19.209447 24430 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:05:19.249838 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 83103.1us
I20251023 02:05:19.248430 24093 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 02:05:19.229151 24426 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:19.274811 23999 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:19.277081 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.280845 22077 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:19.282440 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:19.283601 21961 raft_consensus.cc:2376] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5. Re-sending same reply.
I20251023 02:05:19.256296 24424 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:19.255507 24436 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:19.285374 22077 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:19.291960 22076 leader_election.cc:336] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.905s (SENT)
I20251023 02:05:19.292379 22077 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:19.296001 22077 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.544s (SENT) [suppressed 45 similar messages]
I20251023 02:05:19.301551 22076 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:19.266345 24444 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:19.314337 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:19.265276 24431 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:19.315089 21961 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:19.279652 24359 raft_consensus.cc:2749] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:19.279808 24426 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:19.321390 22076 leader_election.cc:336] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 2.000s (SENT)
I20251023 02:05:19.322041 22076 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:19.323228 22076 leader_election.cc:336] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 2.056s (SENT)
I20251023 02:05:19.323769 22076 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:19.281380 24430 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:19.290578 24416 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:19.294111 23999 raft_consensus.cc:697] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:19.294133 24393 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:19.288555 24093 raft_consensus.cc:697] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:19.296031 24418 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:19.296320 24424 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:19.269222 24213 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:19.336047 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 87734.5us
I20251023 02:05:19.336472 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:19.312244 24290 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:19.338528 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 218344us
I20251023 02:05:19.286259 24339 consensus_queue.cc:1048] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20251023 02:05:19.312088 24346 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.331103 24403 consensus_queue.cc:237] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.526255 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:38996 (request call id 660) took 1560 ms. Trace:
I20251023 02:05:19.451140 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:19.604290 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1524) took 2719 ms. Trace:
I20251023 02:05:19.432302 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:19.604411 22033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1183) took 2689 ms (client timeout 1900 ms). Trace:
I20251023 02:05:19.604985 21961 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:19.476207 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48448 (request call id 14) took 2106 ms. Trace:
I20251023 02:05:19.328519 24450 raft_consensus.cc:2804] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election won for term 5
W20251023 02:05:19.601277 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:19.441326 22108 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20251023 02:05:19.476492 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:19.332666 24436 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:19.606923 22103 raft_consensus.cc:2393] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 7: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:19.604703 21959 rpcz_store.cc:276] 1023 02:05:16.885010 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:16.892831 (+ 7821us) service_pool.cc:225] Handling call
1023 02:05:19.509805 (+2616974us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54002dcdd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:19.604229 (+ 94424us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4476297984}
I20251023 02:05:19.606204 22034 rpcz_store.cc:276] 1023 02:05:17.362122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:17.381403 (+ 19281us) service_pool.cc:225] Handling call
1023 02:05:19.468767 (+2087364us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54002a94d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa933f3 00007f9c64a45bca 00007f9c5dfe9934 00007f9c5dfe98c2 00007f9c5dfe97f1 00007f9c5dfe976c 00007f9c5dfe8a22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:19.468904 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2570670464}
I20251023 02:05:19.575626 22031 rpcz_store.cc:276] 1023 02:05:17.981091 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:17.984196 (+ 3105us) service_pool.cc:225] Handling call
1023 02:05:19.514955 (+1530759us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b540026f054. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa02fe0 00007f9c5faa1aca 00007f9c64a4701e 00007f9c5dfee437 00007f9c5dfee3c2 00007f9c5dfee2f1 00007f9c5dfee26c 00007f9c5dfed522 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:19.526162 (+ 11207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1642374016}
I20251023 02:05:19.334161 24416 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:05:19.605163 22033 rpcz_store.cc:269] 1023 02:05:16.914733 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:16.914904 (+ 171us) service_pool.cc:225] Handling call
1023 02:05:19.513948 (+2599044us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b540040d8d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:19.604316 (+ 90368us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":66574208}
I20251023 02:05:19.452723 24290 consensus_queue.cc:1048] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.165s
I20251023 02:05:19.330523 24426 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:19.390112 24359 raft_consensus.cc:2749] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:19.335305 24393 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:19.389696 24425 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:19.519331 24428 log.cc:927] Time spent T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.289s user 0.001s sys 0.000s
I20251023 02:05:19.334818 24405 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.386619 24457 raft_consensus.cc:2749] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:19.619086 24275 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:19.629259 22036 raft_consensus.cc:1240] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:19.629475 22110 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
W20251023 02:05:19.400381 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 77198.9us
W20251023 02:05:19.630738 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:19.425868 24418 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251023 02:05:19.378221 24443 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:19.601538 24437 raft_consensus.cc:3055] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
W20251023 02:05:19.625339 24441 log.cc:927] Time spent T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.297s user 0.001s sys 0.005s
I20251023 02:05:19.627403 24450 raft_consensus.cc:697] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:19.475817 24353 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:19.635766 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:19.625131 24322 consensus_meta.cc:287] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.754s user 0.006s sys 0.009s
W20251023 02:05:19.632928 22617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34820 (request call id 354) took 1664 ms (client timeout 1612 ms). Trace:
I20251023 02:05:19.488957 24389 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:19.624236 24144 consensus_meta.cc:287] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.686s user 0.014s sys 0.000s
I20251023 02:05:19.613022 24436 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:19.639520 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:19.619433 24416 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:19.628345 24400 raft_consensus.cc:1064] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: attempting to promote NON_VOTER 1ab8b0b58147446d84d901b4ed2ff39e to VOTER
W20251023 02:05:19.615960 24438 log.cc:927] Time spent T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.335s user 0.000s sys 0.006s
W20251023 02:05:19.632015 24448 log.cc:927] Time spent T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.200s user 0.006s sys 0.000s
W20251023 02:05:19.649327 22076 leader_election.cc:336] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 2.197s (SENT)
I20251023 02:05:19.605103 24312 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:05:19.637907 24445 log.cc:927] Time spent T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.148s user 0.000s sys 0.009s
I20251023 02:05:19.629912 24442 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:19.862636 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1578) took 1416 ms. Trace:
I20251023 02:05:19.863173 22102 rpcz_store.cc:276] 1023 02:05:18.446451 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.446601 (+ 150us) service_pool.cc:225] Handling call
1023 02:05:18.447026 (+ 425us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:18.522422 (+ 75396us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:18.522468 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:18.815020 (+292552us) log.cc:844] Serialized 26 byte log entry
1023 02:05:19.100499 (+285479us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:19.100535 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:19.103298 (+ 2763us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:19.862414 (+759116us) raft_consensus.cc:1685] finished
1023 02:05:19.862438 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:19.862571 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":30621,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":104,"thread_start_us":218616,"threads_started":1,"wal-append.queue_time_us":329466}
I20251023 02:05:19.633183 24393 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:19.645402 24427 raft_consensus.cc:697] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:19.676496 21959 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
W20251023 02:05:19.676687 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:19.378400 24368 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:19.681708 22103 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:19.685308 21955 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:19.685560 22076 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:19.502027 23999 consensus_queue.cc:237] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.688158 22031 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:19.514487 24093 consensus_queue.cc:237] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:19.644447 24444 raft_consensus.cc:697] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:19.657588 24437 raft_consensus.cc:740] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:19.646553 24425 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.870426 22002 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:19.870435 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:19.687258 24466 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:19.680029 24322 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:19.681208 24443 raft_consensus.cc:697] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:19.877269 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:19.684180 24429 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:19.688071 24312 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:19.682940 24436 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:19.653764 24275 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:19.900357 22031 raft_consensus.cc:606] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:19.686734 24213 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:19.849992 24331 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:19.876595 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.106848 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:19.922734 21960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1183) took 2646 ms (client timeout 2213 ms). Trace:
W20251023 02:05:20.109892 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:20.116731 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 467431us
I20251023 02:05:20.108351 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:20.117899 22037 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:19.936728 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1450) took 1014 ms. Trace:
W20251023 02:05:20.092772 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:20.118005 21961 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
W20251023 02:05:20.119493 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:20.120703 22102 raft_consensus.cc:1275] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251023 02:05:20.107656 21954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1182) took 2837 ms (client timeout 2197 ms). Trace:
I20251023 02:05:19.951682 22104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1456) took 1027 ms. Trace:
I20251023 02:05:20.108134 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:20.118127 22033 raft_consensus.cc:2393] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
W20251023 02:05:20.119024 22076 leader_election.cc:336] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 2.213s (SENT)
W20251023 02:05:20.121603 21954 rpcz_store.cc:269] 1023 02:05:17.270160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.932539 (+2662379us) service_pool.cc:212] Skipping call since client already timed out
1023 02:05:20.107573 (+175034us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:05:20.123379 22106 rpcz_store.cc:276] 1023 02:05:18.922796 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.938102 (+ 15306us) service_pool.cc:225] Handling call
1023 02:05:18.954309 (+ 16207us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:19.082577 (+128268us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:19.082650 (+ 73us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:19.101801 (+ 19151us) log.cc:844] Serialized 26 byte log entry
1023 02:05:19.208807 (+107006us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:19.208845 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:19.208908 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:19.936432 (+727524us) raft_consensus.cc:1685] finished
1023 02:05:19.936462 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:19.936657 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":38190,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":2563,"thread_start_us":155372,"threads_started":2,"wal-append.queue_time_us":129136}
I20251023 02:05:20.124567 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:19.921911 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.122520 22104 rpcz_store.cc:276] 1023 02:05:18.924608 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.932065 (+ 7457us) service_pool.cc:225] Handling call
1023 02:05:18.932318 (+ 253us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:19.096408 (+164090us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:19.096442 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:19.096487 (+ 45us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:19.096525 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:19.951343 (+854818us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b54000fb254. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:19.951415 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:19.951606 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56767,"raft.queue_time_us":10472,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":15000,"spinlock_wait_cycles":353883008}
W20251023 02:05:20.117897 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 437603us
I20251023 02:05:20.125732 21959 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:20.132398 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1207) took 1050 ms. Trace:
I20251023 02:05:20.133126 21959 rpcz_store.cc:276] 1023 02:05:19.082380 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.878234 (+795854us) service_pool.cc:225] Handling call
1023 02:05:20.132329 (+254095us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:19.922319 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:20.134502 22622 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:19.857810 24365 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:20.142836 22032 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:20.142730 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:20.144060 22104 raft_consensus.cc:1240] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
W20251023 02:05:20.123035 21960 rpcz_store.cc:269] 1023 02:05:17.276666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.922511 (+2645845us) service_pool.cc:212] Skipping call since client already timed out
1023 02:05:19.922643 (+ 132us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 02:05:20.149922 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.150830 21960 raft_consensus.cc:2393] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:05:20.151547 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 281264us
I20251023 02:05:20.152673 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.151661 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:19.938185 24431 consensus_meta.cc:287] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.604s user 0.007s sys 0.000s
I20251023 02:05:20.158951 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1186) took 2801 ms. Trace:
I20251023 02:05:20.159461 21960 rpcz_store.cc:276] 1023 02:05:17.357200 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.149684 (+2792484us) service_pool.cc:225] Handling call
1023 02:05:20.158875 (+ 9191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:20.153399 21954 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:20.160349 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:20.160713 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1210) took 1077 ms. Trace:
I20251023 02:05:20.161170 21954 rpcz_store.cc:276] 1023 02:05:19.083129 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.152403 (+1069274us) service_pool.cc:225] Handling call
1023 02:05:20.160645 (+ 8242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:20.161671 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:20.162379 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.162792 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:19.926831 24455 log.cc:927] Time spent T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.303s user 0.000s sys 0.000s
I20251023 02:05:20.163794 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1554) took 1221 ms. Trace:
I20251023 02:05:19.899693 24418 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:20.166056 22032 raft_consensus.cc:1275] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251023 02:05:20.166410 22002 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:19.907408 24440 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:19.917042 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:20.158694 22624 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:19.939205 24460 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
W20251023 02:05:19.926184 24454 log.cc:927] Time spent T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.300s user 0.000s sys 0.000s
I20251023 02:05:20.184226 22106 raft_consensus.cc:1275] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251023 02:05:20.172799 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:20.185511 21925 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:20.164357 21960 rpcz_store.cc:276] 1023 02:05:18.942589 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.162582 (+1219993us) service_pool.cc:225] Handling call
1023 02:05:20.163744 (+ 1162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:20.189500 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:20.183259 21954 raft_consensus.cc:1240] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20251023 02:05:20.158578 22103 raft_consensus.cc:1275] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251023 02:05:20.190586 22110 raft_consensus.cc:2393] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:20.191533 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1187) took 2485 ms. Trace:
I20251023 02:05:19.938867 24405 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:19.917326 24458 raft_consensus.cc:2749] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:05:19.680416 22617 rpcz_store.cc:269] 1023 02:05:17.969059 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:17.984335 (+ 15276us) service_pool.cc:225] Handling call
1023 02:05:19.508987 (+1524652us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b5400171e54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:19.628782 (+119795us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2081853696}
W20251023 02:05:20.202960 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 83975.1us
W20251023 02:05:20.204214 22076 leader_election.cc:336] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.900s (SENT)
W20251023 02:05:20.205191 22076 leader_election.cc:336] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 2.317s (SENT)
I20251023 02:05:19.997274 24189 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:19.917404 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:20.160369 24346 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:20.041736 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:20.172333 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:20.172211 22105 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:19.878301 24442 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:20.192066 21954 rpcz_store.cc:276] 1023 02:05:17.706709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.182484 (+2475775us) service_pool.cc:225] Handling call
1023 02:05:20.182949 (+ 465us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:20.191200 (+ 8251us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:20.191464 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:20.171365 22037 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20251023 02:05:20.173600 24402 consensus_meta.cc:287] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.947s user 0.010s sys 0.004s
I20251023 02:05:20.185864 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:20.196286 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:20.172127 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.142450 24331 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:20.199805 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:20.267059 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:20.207531 22076 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:20.210309 22108 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.012s sys 0.078s
W20251023 02:05:19.929899 24459 log.cc:927] Time spent T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.250s user 0.000s sys 0.001s
W20251023 02:05:20.188789 24350 consensus_meta.cc:287] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.025s user 0.007s sys 0.008s
I20251023 02:05:20.182296 24213 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:20.173384 24292 consensus_meta.cc:287] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.953s user 0.010s sys 0.007s
I20251023 02:05:20.164631 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.282794 21959 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:20.283280 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1211) took 1199 ms. Trace:
I20251023 02:05:20.283553 21959 rpcz_store.cc:276] 1023 02:05:19.083423 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.134071 (+1050648us) service_pool.cc:225] Handling call
1023 02:05:20.283229 (+149158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:20.210134 24449 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251023 02:05:20.268631 22108 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s user 0.012s sys 0.078s
I20251023 02:05:20.187439 24294 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:20.187680 24295 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:20.331934 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:20.187546 24462 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:20.279114 22106 raft_consensus.cc:1275] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20251023 02:05:20.280290 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 75142.6us
I20251023 02:05:20.278276 21961 raft_consensus.cc:2376] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5. Re-sending same reply.
I20251023 02:05:20.219058 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:20.334271 22076 leader_election.cc:336] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 2.360s (SENT)
I20251023 02:05:20.284782 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:20.319658 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:20.213708 24052 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s user 0.006s sys 0.027s
W20251023 02:05:20.339087 21908 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 98f1206f4e204b72af762bd083d766fd, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:20.339489 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 98f1206f4e204b72af762bd083d766fd, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.339713 24442 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:20.340370 24295 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.345978 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:20.348325 22075 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:20.352214 23717 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:20.353497 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.354152 21954 raft_consensus.cc:2468] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 1.
I20251023 02:05:20.197963 24339 consensus_queue.cc:1059] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20251023 02:05:20.354995 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.355559 21954 raft_consensus.cc:2376] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6. Re-sending same reply.
I20251023 02:05:20.321470 21958 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:20.321480 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:20.358022 22105 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:20.142284 24429 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.145099 24466 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:20.336898 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:20.265779 22620 raft_consensus.cc:2376] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5. Re-sending same reply.
I20251023 02:05:20.210314 24204 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:20.327929 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:20.172741 24431 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.219039 24189 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.336006 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:20.321014 24449 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:20.334277 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1212) took 1128 ms. Trace:
I20251023 02:05:20.434186 21961 rpcz_store.cc:276] 1023 02:05:19.205934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.120269 (+914335us) service_pool.cc:225] Handling call
1023 02:05:20.334226 (+213957us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:19.925992 24427 consensus_queue.cc:237] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:20.369513 21959 raft_consensus.cc:2376] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5. Re-sending same reply.
I20251023 02:05:20.368930 24204 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:20.320113 24458 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:20.106675 24444 consensus_queue.cc:237] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.376262 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:20.490834 22105 raft_consensus.cc:2393] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 6: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:20.490638 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.334375 22035 raft_consensus.cc:2376] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5. Re-sending same reply.
I20251023 02:05:20.377797 22104 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:20.492313 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:20.492985 22104 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:20.107353 24437 consensus_queue.cc:260] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:20.493821 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:20.495679 24437 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:20.497551 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1196) took 2526 ms. Trace:
W20251023 02:05:20.499636 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:20.502858 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:20.424468 23717 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:20.418744 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:20.367167 24426 consensus_meta.cc:287] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.733s user 0.004s sys 0.005s
I20251023 02:05:20.281337 24331 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:20.423132 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:20.423883 21928 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:20.430070 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:19.933956 24450 consensus_queue.cc:237] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.435145 22111 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:20.327059 24213 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.358644 22589 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:20.383081 22618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 241) took 3526 ms (client timeout 2168 ms). Trace:
I20251023 02:05:20.421729 22076 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:20.383755 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:20.158191 24365 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:20.493961 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:20.512764 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 88967.4us
I20251023 02:05:20.378695 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.443151 24449 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:20.423534 24477 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:20.476425 24366 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:20.517098 21928 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:20.500358 21959 rpcz_store.cc:276] 1023 02:05:17.971027 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.438570 (+2467543us) service_pool.cc:225] Handling call
1023 02:05:20.493768 (+ 55198us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:20.495278 (+ 1510us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:20.495434 (+ 156us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:20.495779 (+ 345us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:20.496055 (+ 276us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:20.496506 (+ 451us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:20.497112 (+ 606us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:20.535285 21928 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:20.535429 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:20.514214 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 155629us
W20251023 02:05:20.518070 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 96433.7us
I20251023 02:05:20.490203 24405 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:20.489275 24204 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:20.536682 22105 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20251023 02:05:20.502547 24425 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:20.531033 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.537282 22589 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:20.536784 22076 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:20.546819 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:20.550812 21961 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:20.552819 22076 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:20.553583 24463 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:20.554335 24463 raft_consensus.cc:697] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:20.548884 22037 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:20.511473 24339 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:20.559990 22076 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:20.563098 22076 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:20.519457 24291 raft_consensus.cc:2804] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 02:05:20.512521 24322 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:20.195585 24391 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.373s
I20251023 02:05:20.530248 23717 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:20.570604 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:20.534826 24275 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:20.183735 24440 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.575057 22105 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:20.574205 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:20.561537 24405 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:20.576628 24430 consensus_meta.cc:287] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.187s user 0.008s sys 0.002s
I20251023 02:05:20.579061 21925 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:20.565830 22616 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251023 02:05:20.583222 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:20.568616 24353 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251023 02:05:20.581872 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 151995us
I20251023 02:05:20.567489 24457 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:20.560582 24458 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:20.362298 24476 consensus_queue.cc:1048] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.211s
I20251023 02:05:20.518020 24426 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:20.573985 23999 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:20.566383 24290 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:20.561220 22617 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
W20251023 02:05:20.515772 22618 rpcz_store.cc:269] 1023 02:05:16.841170 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:16.847696 (+ 6526us) service_pool.cc:225] Handling call
1023 02:05:20.355583 (+3507887us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x7b540065ead4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:20.367041 (+ 11458us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2292610304}
I20251023 02:05:20.586393 24481 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:20.593784 22032 raft_consensus.cc:2393] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:20.605659 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:20.644771 22035 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:20.568406 24291 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:20.196424 24460 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:20.607049 24429 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:20.549611 24467 consensus_queue.cc:1048] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.038s
I20251023 02:05:20.182634 24418 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.583216 24275 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:20.619580 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:20.715098 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 152049us
I20251023 02:05:20.622809 24426 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:20.623704 24470 consensus_queue.cc:1059] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20251023 02:05:20.607775 24458 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:20.577324 24479 consensus_queue.cc:1059] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20251023 02:05:20.625866 22622 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9. Ops: []
W20251023 02:05:20.628029 21958 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36488 (request call id 170) took 1817 ms (client timeout 2134 ms). Trace:
I20251023 02:05:20.555481 24463 consensus_queue.cc:237] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:20.364463 24462 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:20.568848 24339 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:20.590386 24353 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:20.608081 24189 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:20.602355 24457 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:20.366513 24443 consensus_queue.cc:237] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.715206 22105 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:20.589510 24292 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:20.557608 24477 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:20.585633 24430 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:20.716008 22076 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:05:21.036978 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 429235us
W20251023 02:05:20.781301 21962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36488 (request call id 165) took 1971 ms (client timeout 1792 ms). Trace:
W20251023 02:05:20.937064 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:21.036706 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:21.250541 21958 rpcz_store.cc:269] 1023 02:05:18.810829 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.222365 (+411536us) service_pool.cc:225] Handling call
1023 02:05:20.627878 (+1405513us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540024bdd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:20.627980 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2506587392}
I20251023 02:05:20.756072 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:21.277827 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 742606us
I20251023 02:05:20.782245 22109 raft_consensus.cc:1275] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 2. (term mismatch)
W20251023 02:05:20.995666 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:21.254592 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:21.278970 22105 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:20.646593 24291 raft_consensus.cc:3060] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 1 FOLLOWER]: Advancing to term 2
W20251023 02:05:21.229874 21961 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.018s sys 0.002s
I20251023 02:05:21.281569 21928 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:21.041312 21926 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.036s sys 0.050s
W20251023 02:05:21.254133 21956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1194) took 3372 ms (client timeout 1900 ms). Trace:
W20251023 02:05:21.280612 21962 rpcz_store.cc:269] 1023 02:05:18.809492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.992604 (+183112us) service_pool.cc:225] Handling call
1023 02:05:20.781119 (+1788515us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b540030f7d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:20.781231 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":335012096}
I20251023 02:05:20.779438 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:21.294515 22618 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
W20251023 02:05:20.713045 24424 consensus_meta.cc:287] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.312s user 0.009s sys 0.002s
I20251023 02:05:21.295751 24424 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:21.295364 24204 consensus_meta.cc:287] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.728s user 0.010s sys 0.000s
I20251023 02:05:21.296373 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1441) took 1235 ms. Trace:
I20251023 02:05:21.296171 24479 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.297107 24479 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.296770 22031 rpcz_store.cc:276] 1023 02:05:20.060959 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.157653 (+ 96694us) service_pool.cc:225] Handling call
1023 02:05:20.184872 (+ 27219us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:20.338108 (+153236us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:20.338142 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:20.365077 (+ 26935us) log.cc:844] Serialized 26 byte log entry
1023 02:05:20.583343 (+218266us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:20.583387 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:20.583455 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:21.296185 (+712730us) raft_consensus.cc:1685] finished
1023 02:05:21.296204 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.296321 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15558,"raft.queue_time_us":164619,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":233228,"threads_started":2,"wal-append.queue_time_us":489818}
I20251023 02:05:21.298854 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:21.299544 22031 raft_consensus.cc:2393] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:21.299655 24479 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.299939 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1214) took 1130 ms. Trace:
I20251023 02:05:21.301070 22031 rpcz_store.cc:276] 1023 02:05:20.169083 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.298611 (+1129528us) service_pool.cc:225] Handling call
1023 02:05:21.299886 (+ 1275us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:21.301777 24501 log.cc:927] Time spent T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.356s user 0.000s sys 0.007s
W20251023 02:05:21.302306 24437 consensus_meta.cc:287] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.806s user 0.012s sys 0.000s
I20251023 02:05:21.302244 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:21.302937 24437 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.303195 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1209) took 1135 ms. Trace:
I20251023 02:05:21.303411 24437 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.303684 22031 rpcz_store.cc:276] 1023 02:05:20.167768 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.301949 (+1134181us) service_pool.cc:225] Handling call
1023 02:05:21.303145 (+ 1196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.304692 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:21.305492 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1206) took 1138 ms. Trace:
I20251023 02:05:20.894805 24442 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.308058 24442 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.309028 24462 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:21.309815 22034 raft_consensus.cc:1240] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20251023 02:05:21.310087 24424 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:20.778437 24416 consensus_meta.cc:287] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.090s user 0.005s sys 0.006s
I20251023 02:05:21.310907 24442 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.271019 24504 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.305522 24437 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:21.252701 24292 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:21.312182 24292 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:21.312328 24504 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.314513 24504 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:20.778226 24213 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:20.645395 24290 raft_consensus.cc:697] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:21.306821 24339 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:21.315476 24339 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:21.315842 24449 consensus_meta.cc:287] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.777s user 0.010s sys 0.000s
I20251023 02:05:21.315991 24339 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.316690 24495 raft_consensus.cc:3055] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:21.316447 24449 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:20.874626 24429 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.317806 24429 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.318506 24339 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:20.656708 24393 consensus_meta.cc:287] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.750s user 0.008s sys 0.002s
I20251023 02:05:21.308621 22028 raft_consensus.cc:1240] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:21.307802 22031 rpcz_store.cc:276] 1023 02:05:20.166979 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.304441 (+1137462us) service_pool.cc:225] Handling call
1023 02:05:21.305447 (+ 1006us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.319885 24429 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:21.255559 24353 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:21.321048 24449 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:20.782115 24418 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.319752 24393 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.322302 21954 raft_consensus.cc:1240] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:21.324389 24393 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.293532 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:21.296778 24204 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.319826 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:20.780053 24478 log.cc:927] Time spent T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.211s user 0.002s sys 0.000s
I20251023 02:05:21.326084 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1448) took 1162 ms. Trace:
I20251023 02:05:21.326426 22031 rpcz_store.cc:276] 1023 02:05:20.164196 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.319572 (+1155376us) service_pool.cc:225] Handling call
1023 02:05:21.326027 (+ 6455us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.311110 24416 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.320624 24290 consensus_queue.cc:237] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.317090 24495 raft_consensus.cc:740] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:21.327256 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:21.335402 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1207) took 1168 ms. Trace:
W20251023 02:05:21.283633 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:21.331218 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.350428 21958 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:21.281417 22109 pending_rounds.cc:85] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 1
W20251023 02:05:21.279917 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:21.338979 21961 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s user 0.020s sys 0.004s
W20251023 02:05:21.341261 21926 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.837s user 0.043s sys 0.053s
W20251023 02:05:21.351395 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 847248us
W20251023 02:05:21.341734 21956 rpcz_store.cc:269] 1023 02:05:17.881368 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.962505 (+1081137us) service_pool.cc:225] Handling call
1023 02:05:21.250159 (+2287654us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b54000c5154. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:21.254060 (+ 3901us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32219904}
I20251023 02:05:21.342880 21925 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:21.352890 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.354969 21956 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
I20251023 02:05:21.346539 22075 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:20.766497 24112 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:20.781824 24295 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:20.906045 24366 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:21.348294 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:21.349238 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1213) took 1180 ms. Trace:
I20251023 02:05:21.348516 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.350873 22109 pending_rounds.cc:107] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251023 02:05:21.346411 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:21.349066 22037 rpcz_store.cc:276] 1023 02:05:20.167252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.321110 (+153858us) service_pool.cc:225] Handling call
1023 02:05:21.335213 (+1014103us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b540026fa54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:21.335346 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":497162112}
W20251023 02:05:21.353461 22077 leader_election.cc:336] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.544s (SENT)
W20251023 02:05:20.922585 24323 consensus_meta.cc:287] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.743s user 0.011s sys 0.000s
I20251023 02:05:21.362459 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:20.756336 24484 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:21.365638 22109 raft_consensus.cc:2889] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:21.365873 22105 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:21.366820 22077 leader_election.cc:336] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.703s (SENT)
I20251023 02:05:21.363932 22032 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:20.782660 24431 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.365437 22031 rpcz_store.cc:276] 1023 02:05:20.168817 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.327017 (+1158200us) service_pool.cc:225] Handling call
1023 02:05:21.349182 (+ 22165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:20.745615 24466 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.413105 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1222) took 1239 ms. Trace:
W20251023 02:05:20.777817 22615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34820 (request call id 348) took 4452 ms (client timeout 1512 ms). Trace:
I20251023 02:05:20.919751 24188 raft_consensus.cc:3055] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:21.369738 22589 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
W20251023 02:05:21.229314 22619 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.004s sys 0.016s
I20251023 02:05:20.778205 23717 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:20.732604 24310 consensus_queue.cc:1048] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.681s
W20251023 02:05:21.413421 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 131921us
I20251023 02:05:21.368202 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1537) took 3389 ms. Trace:
I20251023 02:05:20.656013 24312 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.393306 22106 raft_consensus.cc:1240] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:21.260092 24457 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:20.664520 24436 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.358834 24418 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:21.417500 24339 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:21.424597 22619 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.795s user 0.006s sys 0.018s
I20251023 02:05:21.413568 22032 rpcz_store.cc:276] 1023 02:05:20.173557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.293184 (+1119627us) service_pool.cc:225] Handling call
1023 02:05:21.413046 (+119862us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.392067 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:21.448525 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:21.449414 22032 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:20.620358 23999 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.332474 24353 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:21.420445 21958 raft_consensus.cc:1240] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:21.455876 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1541) took 3460 ms. Trace:
I20251023 02:05:21.456285 21958 rpcz_store.cc:276] 1023 02:05:17.995429 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.365242 (+3369813us) service_pool.cc:225] Handling call
1023 02:05:21.420252 (+ 55010us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.455562 (+ 35310us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.455811 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:20.786480 24365 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:20.790068 24460 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:21.327291 24430 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.428634 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:21.463594 22103 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:21.434273 24477 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.431435 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:21.432061 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 62452.2us
W20251023 02:05:21.421327 22074 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.010s sys 0.011s
W20251023 02:05:21.467501 22074 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.634s user 0.010s sys 0.012s
I20251023 02:05:21.435472 24466 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:21.367197 24468 raft_consensus.cc:3055] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:21.367429 24504 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:21.508399 24504 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:21.508814 24504 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:21.414954 22077 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:21.510159 24507 raft_consensus.cc:2749] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:21.510288 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 143491us
I20251023 02:05:21.391744 24481 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:21.414142 24449 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:21.513993 24449 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:21.514495 24449 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:21.459722 24339 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.370077 24052 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251023 02:05:21.439105 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 181652us
W20251023 02:05:21.436053 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 720158us
I20251023 02:05:21.436380 21928 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.436271 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1470) took 1230 ms. Trace:
W20251023 02:05:21.527547 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 97413.3us
I20251023 02:05:21.441735 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1433) took 2186 ms. Trace:
I20251023 02:05:21.531041 22590 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:21.413792 22109 pending_rounds.cc:107] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251023 02:05:21.533334 22109 raft_consensus.cc:2889] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:21.444939 22105 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:21.412846 24331 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.450086 21957 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
W20251023 02:05:21.526183 22077 leader_election.cc:336] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.734s (SENT)
I20251023 02:05:21.523208 22110 rpcz_store.cc:276] 1023 02:05:20.206285 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.212424 (+ 6139us) service_pool.cc:225] Handling call
1023 02:05:20.630804 (+418380us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:05:20.630990 (+ 186us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:20.631210 (+ 220us) raft_consensus.cc:1534] Early marking committed up to index 5
1023 02:05:21.158117 (+526907us) log.cc:844] Serialized 14 byte log entry
1023 02:05:21.433690 (+275573us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:21.433739 (+ 49us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:21.433783 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.436116 (+ 2333us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.436222 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11,"prepare.queue_time_us":317845,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":218,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":8171,"spinlock_wait_cycles":8103680,"thread_start_us":719269,"threads_started":2,"wal-append.queue_time_us":335597}
I20251023 02:05:21.522063 22076 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:21.542547 22076 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:21.529670 22030 rpcz_store.cc:276] 1023 02:05:19.255696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.255860 (+ 164us) service_pool.cc:225] Handling call
1023 02:05:19.278671 (+ 22811us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:19.433108 (+154437us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:19.433154 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:19.433241 (+ 87us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:19.433299 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.441514 (+2008215us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b540027b354. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:21.441546 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.441675 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":354532736}
I20251023 02:05:21.368033 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1540) took 3378 ms. Trace:
W20251023 02:05:21.419456 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 135765us
I20251023 02:05:21.452754 22037 raft_consensus.cc:2393] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 7: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:21.545105 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1447) took 1394 ms. Trace:
I20251023 02:05:21.545526 22037 rpcz_store.cc:276] 1023 02:05:20.150591 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.391798 (+1241207us) service_pool.cc:225] Handling call
1023 02:05:21.545034 (+153236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.453285 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:21.420423 24188 raft_consensus.cc:740] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:21.458504 22621 raft_consensus.cc:1240] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:21.433125 24189 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.424765 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:21.499724 24353 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.433401 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:21.255337 24458 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:21.464489 22107 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:21.464933 21925 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:21.634091 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:21.392271 24366 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.470939 22035 raft_consensus.cc:2376] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7. Re-sending same reply.
I20251023 02:05:21.367923 24112 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.491168 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:21.435024 24213 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20251023 02:05:21.486656 23717 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.496497 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1212) took 1327 ms. Trace:
W20251023 02:05:21.417328 22615 rpcz_store.cc:269] 1023 02:05:16.325082 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:16.328817 (+ 3735us) service_pool.cc:225] Handling call
1023 02:05:20.766702 (+4437885us) spinlock_profiling.cc:243] Waited 609 ms on lock 0x7b5400181854. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:20.777721 (+ 11019us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1400848128}
I20251023 02:05:21.452344 24418 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.437520 24484 consensus_queue.cc:1059] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20251023 02:05:21.487036 24466 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:21.486943 24468 raft_consensus.cc:740] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:21.513406 22104 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.802s user 0.031s sys 0.042s
I20251023 02:05:21.433936 21961 rpcz_store.cc:276] 1023 02:05:17.978402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.551544 (+2573142us) service_pool.cc:225] Handling call
1023 02:05:20.570612 (+ 19068us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:20.570860 (+ 248us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:20.620611 (+ 49751us) log.cc:844] Serialized 14 byte log entry
1023 02:05:21.367900 (+747289us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:21.367953 (+ 53us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:21.367993 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.368062 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.368152 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16339,"prepare.queue_time_us":357566,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":18016,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":4237,"thread_start_us":776777,"threads_started":2}
W20251023 02:05:21.547646 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 111374us
I20251023 02:05:21.629930 24509 raft_consensus.cc:2749] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:21.690508 24291 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.691597 22624 raft_consensus.cc:1240] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:05:21.692010 22622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 361) took 1481 ms. Trace:
I20251023 02:05:21.692412 22622 rpcz_store.cc:276] 1023 02:05:20.210410 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.347116 (+136706us) service_pool.cc:225] Handling call
1023 02:05:20.347470 (+ 354us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.252479 (+905009us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.691789 (+439310us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b540067e3d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:21.691956 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":639850752}
I20251023 02:05:21.693188 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:21.693544 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:21.695425 24204 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.696399 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:21.630703 24189 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.698791 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1600) took 1423 ms. Trace:
I20251023 02:05:21.698997 24189 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.699204 22102 rpcz_store.cc:276] 1023 02:05:20.275820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.276009 (+ 189us) service_pool.cc:225] Handling call
1023 02:05:21.698108 (+1422099us) spinlock_profiling.cc:243] Waited 3.36 ms on lock 0x7b58002d5798. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:05:21.698294 (+ 186us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.698471 (+ 177us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:21.698498 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:21.698540 (+ 42us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:21.698574 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.698627 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.698737 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7722240}
I20251023 02:05:21.547346 21956 rpcz_store.cc:276] 1023 02:05:17.990087 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.364887 (+3374800us) service_pool.cc:225] Handling call
1023 02:05:21.365202 (+ 315us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.365379 (+ 177us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:21.365402 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:21.365444 (+ 42us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:21.366878 (+ 1434us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.366943 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.367060 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.555249 22000 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.561273 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:21.565369 22076 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:21.612429 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1543) took 3606 ms. Trace:
W20251023 02:05:21.626209 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 55654.9us
I20251023 02:05:21.462030 22002 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:21.702416 21958 rpcz_store.cc:276] 1023 02:05:18.005820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.464996 (+3459176us) service_pool.cc:225] Handling call
1023 02:05:21.492085 (+ 27089us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:21.492494 (+ 409us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:21.492537 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:21.507185 (+ 14648us) log.cc:844] Serialized 26 byte log entry
1023 02:05:21.610418 (+103233us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:21.610463 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.610535 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:21.612165 (+ 1630us) raft_consensus.cc:1685] finished
1023 02:05:21.612184 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.612353 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":20597,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":102792,"threads_started":1,"wal-append.queue_time_us":103136}
I20251023 02:05:21.627813 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:21.705518 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:21.706092 22032 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:21.706789 21958 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:21.707392 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1545) took 3686 ms. Trace:
I20251023 02:05:21.518608 22619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 355) took 1137 ms. Trace:
I20251023 02:05:21.708325 24449 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:21.634459 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 169633us
I20251023 02:05:21.634970 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1211) took 1466 ms. Trace:
I20251023 02:05:21.709259 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1442) took 1624 ms. Trace:
I20251023 02:05:21.709517 22030 rpcz_store.cc:276] 1023 02:05:20.168316 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.559568 (+1391252us) service_pool.cc:225] Handling call
1023 02:05:21.634909 (+ 75341us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.709702 22034 rpcz_store.cc:276] 1023 02:05:20.084529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.158597 (+ 74068us) service_pool.cc:225] Handling call
1023 02:05:20.195896 (+ 37299us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:21.310276 (+1114380us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.709033 (+398757us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54002a7bd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:21.709201 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2561392896}
I20251023 02:05:21.710397 24449 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.540624 24003 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.707674 21958 rpcz_store.cc:276] 1023 02:05:18.020993 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.706340 (+3685347us) service_pool.cc:225] Handling call
1023 02:05:21.706630 (+ 290us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.707164 (+ 534us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.707320 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.711090 24003 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.711921 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:21.711069 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:21.349335 24495 consensus_queue.cc:260] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.540274 24416 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.540642 24144 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.552215 24477 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:21.553552 24275 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:20.779327 24400 consensus_queue.cc:237] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.629364 24507 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:21.641222 22031 rpcz_store.cc:276] 1023 02:05:20.168577 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.413581 (+1245004us) service_pool.cc:225] Handling call
1023 02:05:21.496414 (+ 82833us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:21.727593 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 162345us
I20251023 02:05:21.629020 24431 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:21.651573 21928 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:21.729013 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 267099us
W20251023 02:05:21.652272 22104 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.990s user 0.031s sys 0.043s
I20251023 02:05:21.652770 24366 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.646296 23717 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.729856 24144 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:21.653446 24492 log.cc:927] Time spent T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.164s user 0.000s sys 0.000s
I20251023 02:05:21.725052 24294 consensus_queue.cc:1059] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20251023 02:05:21.654096 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:21.727663 24290 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:21.659052 24508 log.cc:927] Time spent T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20251023 02:05:21.660085 24468 consensus_queue.cc:260] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.667286 24418 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:21.668125 24494 log.cc:927] Time spent T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20251023 02:05:21.731665 24443 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:21.750533 24366 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:21.753023 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:21.753916 24294 raft_consensus.cc:3055] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 02:05:21.727699 24477 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:21.680022 24188 consensus_queue.cc:260] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.682596 24339 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.650466 24466 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:21.685896 24436 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.687009 21961 raft_consensus.cc:1240] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:21.767328 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1199) took 3759 ms. Trace:
I20251023 02:05:21.767802 21961 rpcz_store.cc:276] 1023 02:05:18.008385 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.651101 (+3642716us) service_pool.cc:225] Handling call
1023 02:05:21.686777 (+ 35676us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.766977 (+ 80200us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.767256 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.687906 23999 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.688642 24312 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.688721 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1473) took 1481 ms. Trace:
I20251023 02:05:21.705456 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1423) took 2586 ms. Trace:
I20251023 02:05:21.773109 22036 rpcz_store.cc:276] 1023 02:05:19.118763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.118855 (+ 92us) service_pool.cc:225] Handling call
1023 02:05:19.130727 (+ 11872us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:19.646981 (+516254us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.705165 (+2058184us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b540001d6d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:21.705395 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1140815232}
I20251023 02:05:21.774864 22036 raft_consensus.cc:3055] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:21.710726 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:21.720006 22032 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
W20251023 02:05:21.721294 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 166167us
I20251023 02:05:21.716655 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:21.722543 21957 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1214) took 2484 ms (client timeout 1544 ms). Trace:
I20251023 02:05:21.775991 22105 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:21.776398 22032 raft_consensus.cc:1240] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251023 02:05:21.733525 23717 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:21.713323 24003 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.728420 22076 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:21.513994 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:21.730233 22104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 237) took 1364 ms. Trace:
I20251023 02:05:21.637758 24052 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:21.727478 24509 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:21.738130 22590 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:21.738334 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 86846.9us
I20251023 02:05:21.780725 21925 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:21.781843 21928 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.732152 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:21.754361 24294 raft_consensus.cc:740] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:21.739871 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:21.783499 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 45447.7us
I20251023 02:05:21.784008 22102 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:21.735435 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.745874 22034 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:21.736150 22001 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:21.785035 22034 raft_consensus.cc:606] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:21.785223 21958 raft_consensus.cc:2410] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20251023 02:05:21.785687 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48448 (request call id 16) took 1567 ms. Trace:
I20251023 02:05:21.785856 24295 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:21.786075 22034 rpcz_store.cc:276] 1023 02:05:20.218571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.713684 (+1495113us) service_pool.cc:225] Handling call
1023 02:05:21.785624 (+ 71940us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.786582 24439 raft_consensus.cc:2749] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20251023 02:05:21.735266 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:21.679198 24112 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:21.785411 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 49333.8us
I20251023 02:05:21.787748 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:21.745399 22031 raft_consensus.cc:1240] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:21.711045 24449 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.787073 24350 raft_consensus.cc:3055] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:21.725191 24310 consensus_queue.cc:1059] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20251023 02:05:21.729959 24507 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:21.743455 24290 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.734862 24431 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.761511 24291 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.763203 24477 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:21.734670 24437 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:21.766278 24188 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:21.773051 22111 rpcz_store.cc:276] 1023 02:05:20.207176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.208105 (+ 929us) service_pool.cc:225] Handling call
1023 02:05:20.208363 (+ 258us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:20.512861 (+304498us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.688478 (+1175617us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54000fbc54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:21.688661 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":521014144}
W20251023 02:05:21.776181 21957 rpcz_store.cc:269] 1023 02:05:19.238109 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.254467 (+ 16358us) service_pool.cc:225] Handling call
1023 02:05:21.722411 (+2467944us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b5400107f54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:21.722489 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4859443456}
I20251023 02:05:21.777343 22000 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.723121 21962 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
W20251023 02:05:21.777343 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 49007.4us
I20251023 02:05:21.811739 22000 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.811877 22076 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:21.813274 22000 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:21.724076 21956 raft_consensus.cc:1240] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:21.814970 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1544) took 3804 ms. Trace:
I20251023 02:05:21.815044 22000 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.815390 21956 rpcz_store.cc:276] 1023 02:05:18.010391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.720506 (+3710115us) service_pool.cc:225] Handling call
1023 02:05:21.720762 (+ 256us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.814754 (+ 93992us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.814922 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.775164 22036 raft_consensus.cc:740] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:21.777884 22104 rpcz_store.cc:276] 1023 02:05:20.365712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.531168 (+165456us) service_pool.cc:225] Handling call
1023 02:05:20.532261 (+ 1093us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:20.532755 (+ 494us) raft_consensus.cc:1534] Early marking committed up to index 7
1023 02:05:20.661998 (+129243us) log.cc:844] Serialized 14 byte log entry
1023 02:05:21.729918 (+1067920us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:21.729963 (+ 45us) raft_consensus.cc:1645] Marking committed up to 7
1023 02:05:21.730009 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.730073 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.730181 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":415929,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":58,"raft.queue_time_us":87,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":72,"thread_start_us":1196275,"threads_started":2,"wal-append.queue_time_us":849296}
I20251023 02:05:21.816334 22000 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.778460 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1223) took 1596 ms. Trace:
I20251023 02:05:21.782685 22110 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 3.
I20251023 02:05:21.786314 22030 raft_consensus.cc:1240] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:21.788472 21961 raft_consensus.cc:3060] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:21.816983 22032 rpcz_store.cc:276] 1023 02:05:20.182533 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.775847 (+1593314us) service_pool.cc:225] Handling call
1023 02:05:21.776150 (+ 303us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:21.776894 (+ 744us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.778404 (+ 1510us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.817730 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1450) took 1623 ms. Trace:
I20251023 02:05:21.818079 22030 rpcz_store.cc:276] 1023 02:05:20.194358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.780667 (+1586309us) service_pool.cc:225] Handling call
1023 02:05:21.786155 (+ 5488us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.817494 (+ 31339us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.817681 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.789407 24514 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:21.756364 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:21.729104 24213 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:21.832603 24213 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:21.833020 24213 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:21.833395 22623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 244) took 2181 ms. Trace:
W20251023 02:05:21.834503 24474 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 1.330s user 0.012s sys 0.011s
I20251023 02:05:21.753926 22615 raft_consensus.cc:2393] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:21.789458 22590 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:21.843449 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 54073.9us
I20251023 02:05:21.843559 22000 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:21.790369 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:21.844944 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:21.791325 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1220) took 1620 ms. Trace:
W20251023 02:05:21.793129 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:21.795022 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251023 02:05:21.845357 22031 rpcz_store.cc:276] 1023 02:05:20.170944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.728117 (+1557173us) service_pool.cc:225] Handling call
1023 02:05:21.745277 (+ 17160us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.791152 (+ 45875us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.791285 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:21.780874 24468 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:21.800001 24507 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:21.780578 24052 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:21.783450 24339 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:21.783565 24436 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:21.803982 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.835873 24418 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:21.862931 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.826224 24460 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:21.791013 24003 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:21.824776 24452 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:21.816349 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.877503 21956 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:21.877708 21961 raft_consensus.cc:2468] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:05:21.820591 22587 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.822436 22032 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251023 02:05:21.783171 23999 consensus_queue.cc:1059] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20251023 02:05:21.825186 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:21.781255 24292 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:21.826107 22076 connection.cc:232] RPC call timeout handler was delayed by -1.3142519410000659s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
W20251023 02:05:21.883404 22076 connection.cc:232] RPC call timeout handler was delayed by -1.2858155280000574s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1544s for a response
I20251023 02:05:21.885087 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:21.885982 22104 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:21.887552 22076 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:21.792516 24350 raft_consensus.cc:740] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:21.890357 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:21.831737 22102 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:21.891789 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:21.892635 22102 raft_consensus.cc:2393] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 6: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:21.837333 22103 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:21.894851 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:21.895534 22110 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:05:21.842165 22035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1221) took 1671 ms (client timeout 1900 ms). Trace:
W20251023 02:05:21.896306 22035 rpcz_store.cc:269] 1023 02:05:20.171169 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.655388 (+484219us) service_pool.cc:225] Handling call
1023 02:05:21.841975 (+1186587us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b540040d8d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:21.842094 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":437148032}
W20251023 02:05:21.798316 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:21.850119 24431 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.836761 23717 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.851413 24497 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.900381 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:21.851846 21953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1193) took 3970 ms (client timeout 1926 ms). Trace:
I20251023 02:05:21.851909 24503 consensus_queue.cc:1059] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20251023 02:05:21.852839 24291 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:21.824759 24429 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
W20251023 02:05:21.906347 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 57445us
I20251023 02:05:21.853747 21960 raft_consensus.cc:1240] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:21.862741 24418 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:21.913319 24418 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:21.913640 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:21.914574 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:21.914753 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:21.836993 24430 raft_consensus.cc:2749] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20251023 02:05:21.789423 24469 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.241s user 0.015s sys 0.027s
W20251023 02:05:21.918002 24469 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.383s user 0.016s sys 0.029s
I20251023 02:05:21.788487 21928 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.918591 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1219) took 1747 ms. Trace:
W20251023 02:05:21.919317 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 130908us
I20251023 02:05:21.839600 24514 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.919955 24391 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:21.920040 21928 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.920486 24391 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:21.920946 24391 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:21.921237 24514 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.921774 24275 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20251023 02:05:21.922811 24473 log.cc:927] Time spent T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 1.414s user 0.006s sys 0.005s
I20251023 02:05:21.919044 22037 rpcz_store.cc:276] 1023 02:05:20.170718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.714857 (+1544139us) service_pool.cc:225] Handling call
1023 02:05:21.715167 (+ 310us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:21.726954 (+ 11787us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:21.726974 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:21.730573 (+ 3599us) log.cc:844] Serialized 26 byte log entry
1023 02:05:21.781731 (+ 51158us) raft_consensus.cc:1645] Marking committed up to 8
1023 02:05:21.870356 (+ 88625us) log.cc:844] Serialized 14 byte log entry
1023 02:05:21.870540 (+ 184us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.870614 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:21.918369 (+ 47755us) raft_consensus.cc:1685] finished
1023 02:05:21.918403 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.918519 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14768,"raft.queue_time_us":50847,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":4163,"thread_start_us":103861,"threads_started":2,"wal-append.queue_time_us":81610}
I20251023 02:05:21.924503 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1468) took 2308 ms. Trace:
I20251023 02:05:21.924830 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:21.925642 22105 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:21.875128 24204 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:21.878929 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 58410.1us
I20251023 02:05:21.833854 22623 rpcz_store.cc:276] 1023 02:05:19.652512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.654094 (+ 1582us) service_pool.cc:225] Handling call
1023 02:05:19.878188 (+224094us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:20.569140 (+690952us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:20.570139 (+ 999us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:20.577457 (+ 7318us) raft_consensus.cc:1645] Marking committed up to 8
1023 02:05:20.580554 (+ 3097us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.833154 (+1252600us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b5400600c54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:21.833205 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.833340 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1563098496}
I20251023 02:05:21.806562 24504 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.854173 24439 consensus_queue.cc:1059] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20251023 02:05:21.895443 22104 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 9.
I20251023 02:05:21.853546 24309 raft_consensus.cc:2749] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:21.899227 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:21.901861 21953 rpcz_store.cc:269] 1023 02:05:17.881055 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:18.962730 (+1081675us) service_pool.cc:225] Handling call
1023 02:05:21.851625 (+2888895us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b540030f2d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:21.851774 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":537315712}
I20251023 02:05:21.902235 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:21.950843 21961 raft_consensus.cc:2393] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:21.951113 22102 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:21.853119 24481 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:21.839053 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:21.864132 24339 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:21.857024 24052 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:21.851467 22617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 250) took 1724 ms (client timeout 1782 ms). Trace:
I20251023 02:05:21.863476 24436 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:21.789999 24449 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.898309 24458 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.897724 24431 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:21.909183 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:21.959560 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:21.960685 21957 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:21.910089 24292 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.906179 23717 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.708993 22619 rpcz_store.cc:276] 1023 02:05:20.369212 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.430674 (+ 61462us) service_pool.cc:225] Handling call
1023 02:05:20.562106 (+131432us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:20.577249 (+ 15143us) raft_consensus.cc:1534] Early marking committed up to index 5
1023 02:05:20.618021 (+ 40772us) log.cc:844] Serialized 14 byte log entry
1023 02:05:21.470036 (+852015us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:21.489954 (+ 19918us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:21.491709 (+ 1755us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:21.497289 (+ 5580us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.506190 (+ 8901us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":29965,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":11896,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":2048,"thread_start_us":822257,"threads_started":1,"wal-append.queue_time_us":638513}
W20251023 02:05:21.911329 24475 log.cc:927] Time spent T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 1.362s user 0.008s sys 0.012s
I20251023 02:05:21.836171 24353 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:21.876010 24460 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:21.967921 24353 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.968719 24458 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.969978 24353 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.970474 24449 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:21.921291 21928 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.952795 24481 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:21.924865 22108 rpcz_store.cc:276] 1023 02:05:19.615879 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.616161 (+ 282us) service_pool.cc:225] Handling call
1023 02:05:19.630532 (+ 14371us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:19.637570 (+ 7038us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:19.637615 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:19.643454 (+ 5839us) log.cc:844] Serialized 26 byte log entry
1023 02:05:20.321215 (+677761us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:20.321256 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:20.321318 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:21.922176 (+1600858us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:21.924240 (+ 2064us) raft_consensus.cc:1685] finished
1023 02:05:21.924262 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.924447 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3477,"raft.queue_time_us":15461,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":23750,"spinlock_wait_cycles":133066496,"thread_start_us":633947,"threads_started":1,"wal-append.queue_time_us":693807}
I20251023 02:05:21.933248 21956 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:21.874567 24416 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:21.976909 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:21.871690 21925 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:21.898866 24426 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:21.939654 22587 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:21.925148 24495 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:05:21.880383 22077 leader_election.cc:336] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.665s (SENT)
I20251023 02:05:21.939891 22002 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:21.942605 24430 consensus_queue.cc:1059] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
W20251023 02:05:21.945809 21955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1208) took 2863 ms (client timeout 1859 ms). Trace:
W20251023 02:05:21.947525 24471 log.cc:927] Time spent T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 1.519s user 0.010s sys 0.001s
I20251023 02:05:21.947906 24509 raft_consensus.cc:2749] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:21.839403 24294 consensus_queue.cc:260] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:21.983474 24312 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:21.983721 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 62513.9us
I20251023 02:05:21.909447 24345 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:21.989980 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:21.985677 24449 raft_consensus.cc:2749] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:21.986502 23717 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:21.986799 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.996222 21957 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:21.997071 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:21.926714 24457 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.997707 21957 raft_consensus.cc:2393] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:21.998132 24290 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.998435 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:21.999048 24457 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:21.983417 24416 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.001530 24416 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:21.956180 22104 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251023 02:05:21.961141 22110 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:21.959949 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.934900 24291 raft_consensus.cc:697] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:21.962426 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.963126 22111 raft_consensus.cc:2393] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 6: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:21.889389 24350 consensus_queue.cc:260] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:21.967103 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 118197us
I20251023 02:05:21.968323 24460 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:21.955402 24339 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:21.931253 22103 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:21.973286 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:21.954762 22617 rpcz_store.cc:269] 1023 02:05:20.127243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.281906 (+154663us) service_pool.cc:225] Handling call
1023 02:05:21.851268 (+1569362us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b54004f1d54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:21.851408 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":66349824}
W20251023 02:05:21.976727 21908 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6869bc9cdea241aaa6d86d46a1b5bcfb, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:22.029017 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6869bc9cdea241aaa6d86d46a1b5bcfb, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.906440 24497 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:21.958868 24436 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:21.982908 22587 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.032270 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:22.032595 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 49843.3us
I20251023 02:05:22.033556 22587 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.034159 22108 raft_consensus.cc:1240] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:22.035161 22587 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:21.989390 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1434) took 2709 ms. Trace:
I20251023 02:05:22.036410 24504 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:22.036581 24426 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:22.037242 24426 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:22.037762 24331 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Leader election won for term 9
I20251023 02:05:22.038247 24331 raft_consensus.cc:697] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:22.035173 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:22.042559 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:21.984140 21955 rpcz_store.cc:269] 1023 02:05:19.082635 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.212521 (+129886us) service_pool.cc:225] Handling call
1023 02:05:21.945634 (+2733113us) spinlock_profiling.cc:243] Waited 444 ms on lock 0x7b5400107054. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:21.945743 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1020301056}
I20251023 02:05:21.990576 21928 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.043455 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:22.043629 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 53146.1us
W20251023 02:05:21.983999 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 44218us
W20251023 02:05:21.983736 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 103404us
I20251023 02:05:22.017051 21961 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:22.020569 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.021811 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.027817 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.988351 24430 consensus_queue.cc:1243] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20251023 02:05:22.049002 21956 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:22.050194 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:21.974399 22105 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251023 02:05:22.051184 22002 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:21.942341 24112 consensus_queue.cc:1059] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
W20251023 02:05:22.029937 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 171709us
W20251023 02:05:22.052968 22077 leader_election.cc:336] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.900s (SENT)
I20251023 02:05:22.057015 22077 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:22.060611 22589 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
I20251023 02:05:22.036021 22029 rpcz_store.cc:276] 1023 02:05:19.279978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.288612 (+ 8634us) service_pool.cc:225] Handling call
1023 02:05:19.313674 (+ 25062us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:19.601795 (+288121us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:19.601826 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:19.602319 (+ 493us) log.cc:844] Serialized 26 byte log entry
1023 02:05:19.895208 (+292889us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:19.895270 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:19.895382 (+ 112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:21.946972 (+2051590us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:21.989119 (+ 42147us) raft_consensus.cc:1685] finished
1023 02:05:21.989144 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:21.989331 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":297843,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":9099,"spinlock_wait_cycles":355777792,"thread_start_us":439267,"threads_started":2,"wal-append.queue_time_us":655899}
I20251023 02:05:21.986409 24524 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:22.074635 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.047396 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:21.987826 24440 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:22.067369 22002 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:05:22.075189 22077 leader_election.cc:336] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.926s (SENT)
W20251023 02:05:22.076359 22077 leader_election.cc:336] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.720s (SENT)
I20251023 02:05:22.076476 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:22.077351 21961 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:22.077201 22077 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:22.069509 22076 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:22.046285 22104 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:22.079667 22076 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:22.068702 22110 raft_consensus.cc:2410] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251023 02:05:22.080133 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:22.081260 22076 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:22.039397 24400 raft_consensus.cc:2804] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251023 02:05:22.089287 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.095359 22108 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:22.098701 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.084918 21928 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.100507 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.102128 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.094425 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.105611 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.051607 24429 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:22.049871 24529 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:22.107821 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.108752 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.108814 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.109467 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.109280 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:22.111006 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 43808.6us
I20251023 02:05:22.114674 22031 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:22.115126 22002 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:22.115465 22104 raft_consensus.cc:2393] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 6: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:22.114190 22032 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:22.116423 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.075003 24193 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:22.120422 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.074270 24524 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:22.127786 21961 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:22.128474 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.128907 21961 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
W20251023 02:05:22.121626 21928 connection.cc:232] RPC call timeout handler was delayed by -1.3425507170001083s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
I20251023 02:05:22.120811 21953 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:22.129985 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.131244 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.095407 24292 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:22.094274 24457 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:22.085919 24416 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:22.074220 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:22.088343 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:22.062523 24469 consensus_queue.cc:1048] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.062s
I20251023 02:05:22.087074 24350 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:21.989466 24463 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.065s
I20251023 02:05:22.153537 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:22.153815 22615 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 LEARNER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251023 02:05:22.123888 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.133009 24193 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.155308 24418 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.156566 24193 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:22.103490 24400 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:22.104074 24514 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:22.157598 24400 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:22.143013 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.159559 22104 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:22.027923 24444 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.051s
I20251023 02:05:22.130873 24426 raft_consensus.cc:2804] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 FOLLOWER]: Leader election won for term 2
W20251023 02:05:22.134328 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 99235.3us
I20251023 02:05:22.087330 24504 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:22.116246 24449 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:22.148996 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.133082 24467 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:22.150796 22621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 353) took 2243 ms. Trace:
I20251023 02:05:22.170984 22621 rpcz_store.cc:276] 1023 02:05:19.906992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.907116 (+ 124us) service_pool.cc:225] Handling call
1023 02:05:19.907426 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.548566 (+1641140us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:22.148184 (+599618us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400171e54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:22.150719 (+ 2535us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3566932224}
I20251023 02:05:22.172809 24497 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:22.105528 24456 consensus_queue.cc:1059] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20251023 02:05:22.174017 24456 raft_consensus.cc:3055] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:22.130695 24331 consensus_queue.cc:237] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.174381 24456 raft_consensus.cc:740] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:22.175191 24456 consensus_queue.cc:260] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.176307 24456 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:22.149847 24291 consensus_queue.cc:237] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.095055 24430 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:22.121763 24503 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:22.131168 24524 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:22.164177 22587 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:22.166365 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.167482 22031 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:22.150298 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.108825 24389 consensus_queue.cc:1059] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20251023 02:05:22.144104 24429 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:22.151916 22617 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:22.122035 24529 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.161368 24443 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:22.159801 24418 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:22.164654 24457 raft_consensus.cc:697] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:22.181506 24503 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:22.169288 24213 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:22.259673 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:22.180065 24403 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.611s user 0.012s sys 0.010s
I20251023 02:05:22.187997 22104 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:22.178205 24467 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:22.270493 24467 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:22.177829 24426 raft_consensus.cc:697] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [term 2 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
W20251023 02:05:22.250978 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 86873.7us
I20251023 02:05:22.254956 21925 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:22.255311 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.120460 24440 consensus_queue.cc:1059] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20251023 02:05:22.187852 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:22.261160 22618 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251023 02:05:22.261962 22029 raft_consensus.cc:2376] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7. Re-sending same reply.
I20251023 02:05:22.263233 24529 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:22.273718 24523 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:22.278265 24523 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:22.278618 24523 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:22.266983 24213 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:22.267510 22032 raft_consensus.cc:2368] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6.
I20251023 02:05:22.177539 24449 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.274876 22031 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:22.254570 24360 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:22.269068 24403 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.712s user 0.012s sys 0.010s
I20251023 02:05:22.268006 24503 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:22.284936 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.275188 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.186026 24430 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
W20251023 02:05:22.281975 22077 leader_election.cc:336] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.664s (SENT)
I20251023 02:05:22.267874 24463 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:22.297668 22104 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:22.281008 24476 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:22.271603 24525 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:22.302045 22104 raft_consensus.cc:1240] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:22.286924 22587 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:22.302515 21925 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:22.304867 21959 raft_consensus.cc:1240] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:22.257881 24429 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.313261 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:22.317157 24531 log.cc:927] Time spent T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
W20251023 02:05:22.317852 24528 log.cc:927] Time spent T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.198s user 0.001s sys 0.000s
I20251023 02:05:22.318740 22110 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:22.320223 22076 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:22.322728 24331 raft_consensus.cc:2804] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:22.323278 24331 raft_consensus.cc:697] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:22.324131 24331 consensus_queue.cc:237] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.318908 24476 raft_consensus.cc:697] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:22.330713 24476 consensus_queue.cc:237] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.322132 22076 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:22.307096 24463 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:22.342685 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.343727 22031 raft_consensus.cc:2393] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:22.304811 24292 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20251023 02:05:22.350353 21961 raft_consensus.cc:2410] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20251023 02:05:22.350936 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 57934.9us
W20251023 02:05:22.351085 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:22.351817 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:22.332449 24514 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:22.336175 24525 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:22.295162 24532 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20251023 02:05:22.337447 24440 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:22.344674 22618 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20251023 02:05:22.341841 24463 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:22.357079 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.361294 24003 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:22.362124 22624 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:22.299680 24360 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.367183 24449 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:22.368779 24468 consensus_queue.cc:1059] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251023 02:05:22.369477 24468 consensus_queue.cc:1243] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer 1ab8b0b58147446d84d901b4ed2ff39e log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20251023 02:05:22.369776 24366 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:22.370318 24366 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:22.385390 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:22.364562 21928 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:22.353943 24514 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:22.356839 24440 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:22.393087 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:22.393555 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:22.354970 24525 raft_consensus.cc:697] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
W20251023 02:05:22.404228 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:22.400662 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.406175 22031 raft_consensus.cc:2410] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20251023 02:05:22.372499 24477 consensus_meta.cc:287] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.546s user 0.005s sys 0.005s
I20251023 02:05:22.352911 24426 consensus_queue.cc:237] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.372967 24481 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.398538 24514 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:22.422226 24477 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.481621 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:22.509675 22030 raft_consensus.cc:1240] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:22.468032 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.492120 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.531906 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.531817 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.358197 24457 consensus_queue.cc:237] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.480646 24339 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.533221 22104 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:22.537171 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.537992 21925 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:22.487309 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 127741us
W20251023 02:05:22.405340 24466 consensus_meta.cc:287] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.629s user 0.007s sys 0.003s
W20251023 02:05:22.422017 24391 consensus_meta.cc:287] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.501s user 0.011s sys 0.000s
I20251023 02:05:22.538751 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:22.404203 24440 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:22.546402 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.545113 22104 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:22.547860 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:22.478917 24536 log.cc:927] Time spent T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20251023 02:05:22.415926 24456 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20251023 02:05:22.536795 24312 consensus_meta.cc:287] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.547s user 0.010s sys 0.000s
W20251023 02:05:22.486816 24534 log.cc:927] Time spent T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
I20251023 02:05:22.488556 24366 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:22.551366 21956 raft_consensus.cc:2393] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 8: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:22.557934 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1539) took 4574 ms. Trace:
I20251023 02:05:22.528705 24477 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:22.544415 24495 consensus_meta.cc:287] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.551s user 0.009s sys 0.001s
I20251023 02:05:22.553541 22624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 363) took 1127 ms. Trace:
I20251023 02:05:22.571259 22624 rpcz_store.cc:276] 1023 02:05:21.426282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.694096 (+267814us) service_pool.cc:225] Handling call
1023 02:05:21.854394 (+160298us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:22.362440 (+508046us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:22.363128 (+ 688us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b54004f1d54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:22.553295 (+190167us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54004f1d50. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:22.553465 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1535343232}
I20251023 02:05:22.572238 22104 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:22.572661 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:22.573843 22104 raft_consensus.cc:1275] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20251023 02:05:22.574378 22104 pending_rounds.cc:85] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 2
I20251023 02:05:22.574574 22104 pending_rounds.cc:107] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251023 02:05:22.575243 22104 raft_consensus.cc:2889] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:22.573899 22587 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:22.556783 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1437) took 2649 ms. Trace:
I20251023 02:05:22.557461 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1535) took 4586 ms. Trace:
I20251023 02:05:22.576375 21960 rpcz_store.cc:276] 1023 02:05:17.970732 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.202357 (+2231625us) service_pool.cc:225] Handling call
1023 02:05:21.853561 (+1651204us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.906989 (+ 53428us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:22.557240 (+650251us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b540030f2d0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:22.557393 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3796971264}
I20251023 02:05:22.538784 24469 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:22.552502 24312 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:22.576282 22028 rpcz_store.cc:276] 1023 02:05:19.907918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.908178 (+ 260us) service_pool.cc:225] Handling call
1023 02:05:20.108680 (+200502us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.319446 (+1210766us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:22.556521 (+1237075us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540020bcd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:22.556714 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2759320960}
I20251023 02:05:22.579638 21958 raft_consensus.cc:2410] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20251023 02:05:22.541949 24430 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:22.566557 24405 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:22.548413 24391 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.550644 24525 consensus_queue.cc:237] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:22.581326 24389 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:22.560722 21954 rpcz_store.cc:276] 1023 02:05:17.983205 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.585025 (+2601820us) service_pool.cc:225] Handling call
1023 02:05:20.615788 (+ 30763us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.322670 (+706882us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:22.557693 (+1235023us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b540017c354. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:22.557886 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1624499584}
I20251023 02:05:22.578475 21957 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:22.580312 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.583210 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.584460 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.586649 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.588586 24535 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:22.605732 22111 raft_consensus.cc:1240] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20251023 02:05:22.607188 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1475) took 2247 ms. Trace:
I20251023 02:05:22.609459 22587 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.581024 24425 consensus_queue.cc:1059] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20251023 02:05:22.579716 24469 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:22.614413 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:22.594236 24360 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:22.656107 24535 raft_consensus.cc:697] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:22.599135 22618 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:22.634670 22105 raft_consensus.cc:1240] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:22.657143 24535 consensus_queue.cc:237] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.629974 24514 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.632387 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1542) took 4631 ms. Trace:
I20251023 02:05:22.658574 21962 rpcz_store.cc:276] 1023 02:05:18.000764 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.367653 (+3366889us) service_pool.cc:225] Handling call
1023 02:05:21.414030 (+ 46377us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.811292 (+397262us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:22.632146 (+820854us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b5400024ed4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:22.632332 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":693015296}
I20251023 02:05:22.659696 24514 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:22.595400 24295 raft_consensus.cc:3055] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Stepping down as leader of term 8
I20251023 02:05:22.634393 22077 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:22.618927 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:22.651983 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.632858 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1463) took 3235 ms. Trace:
I20251023 02:05:22.661831 22029 raft_consensus.cc:2393] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:22.655486 22106 rpcz_store.cc:276] 1023 02:05:20.359385 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.518981 (+159596us) service_pool.cc:225] Handling call
1023 02:05:20.558380 (+ 39399us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.434515 (+876135us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:22.606973 (+1172458us) spinlock_profiling.cc:243] Waited 835 ms on lock 0x7b540025c1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:22.607130 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1919634304}
W20251023 02:05:22.662303 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 52954.7us
I20251023 02:05:22.627161 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.661469 22077 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:22.641135 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.660359 24469 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:22.657323 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:22.665691 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.666414 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:22.666922 22111 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 6
I20251023 02:05:22.602905 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:22.659631 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:22.667876 22110 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:22.661722 22107 rpcz_store.cc:276] 1023 02:05:19.397017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:19.398394 (+ 1377us) service_pool.cc:225] Handling call
1023 02:05:19.398695 (+ 301us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.634274 (+2235579us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:22.632661 (+998387us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b540022ef54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:22.632806 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4750753664}
I20251023 02:05:22.668227 21962 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:22.615700 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1536) took 4644 ms. Trace:
I20251023 02:05:22.630864 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:22.641479 22031 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:22.617512 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.674826 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.675464 22032 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:22.676390 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.671536 21926 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.671290 21959 rpcz_store.cc:276] 1023 02:05:17.971044 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.535580 (+2564536us) service_pool.cc:225] Handling call
1023 02:05:20.551076 (+ 15496us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:22.309378 (+1758302us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:22.615484 (+306106us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b5400218754. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:22.615650 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4033152896}
I20251023 02:05:22.656175 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.680471 21956 raft_consensus.cc:2410] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20251023 02:05:22.665304 22077 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:22.677489 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.684966 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.686471 22000 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.687264 21926 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.688854 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.689522 22028 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:22.691627 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.693135 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.693918 21956 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:22.694912 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.665553 24416 raft_consensus.cc:2749] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:22.695930 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.696420 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:22.696733 24052 consensus_meta.cc:287] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.736s user 0.010s sys 0.000s
I20251023 02:05:22.697055 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:22.697598 24347 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:22.697817 22102 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:22.697947 22106 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251023 02:05:22.548903 24466 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.699185 22106 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:22.660463 24295 raft_consensus.cc:740] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:22.705155 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:22.568535 24532 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.100s
W20251023 02:05:22.589931 24436 consensus_meta.cc:287] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.535s user 0.005s sys 0.004s
I20251023 02:05:22.598809 24339 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:22.629096 24391 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:22.618468 22617 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:22.703343 22076 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:22.697175 24052 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.596308 22036 consensus_queue.cc:260] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.683048 24514 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:22.747130 21959 raft_consensus.cc:2393] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:22.750573 22028 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:22.823812 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.746883 24393 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:22.706329 21928 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.748797 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.831991 22032 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:22.629981 24389 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.700520 22107 raft_consensus.cc:1275] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:05:22.638448 24310 raft_consensus.cc:3055] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:22.605091 24524 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.768739 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.842583 22029 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
W20251023 02:05:22.843147 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 136900us
I20251023 02:05:22.843391 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.843947 22029 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:22.703441 22000 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.844874 22111 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
W20251023 02:05:22.845270 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 141901us
I20251023 02:05:22.699911 24543 consensus_queue.cc:1059] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.145s
I20251023 02:05:22.683023 24483 consensus_queue.cc:1059] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.145s
I20251023 02:05:22.847365 22000 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.623160 24144 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:22.807171 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 103936us
I20251023 02:05:22.725097 22102 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:22.807286 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:22.857121 21954 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:22.807832 22621 raft_consensus.cc:1240] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:22.637305 24429 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:22.609539 24467 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.656905 24425 raft_consensus.cc:3055] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:22.650060 24405 consensus_queue.cc:1059] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.149s
I20251023 02:05:22.610373 24437 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:22.706358 24497 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:22.705364 24323 consensus_queue.cc:1059] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.101s
I20251023 02:05:22.825238 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.828574 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1485) took 1482 ms. Trace:
I20251023 02:05:22.713246 21962 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:22.830472 22110 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:22.749825 24368 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:22.833400 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:22.898802 21959 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:22.752610 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.750054 24290 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:22.901196 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1472) took 2694 ms. Trace:
I20251023 02:05:22.901568 22109 rpcz_store.cc:276] 1023 02:05:20.206880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.207514 (+ 634us) service_pool.cc:225] Handling call
1023 02:05:20.207770 (+ 256us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:21.550164 (+1342394us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:22.900982 (+1350818us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b54001dc254. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:22.901144 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1320399232}
I20251023 02:05:22.903014 22109 raft_consensus.cc:1240] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251023 02:05:22.796006 24360 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:22.918031 24440 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:22.918646 24460 consensus_meta.cc:287] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.887s user 0.010s sys 0.000s
I20251023 02:05:22.919090 24460 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:22.919770 24439 raft_consensus.cc:3055] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:22.920229 24439 raft_consensus.cc:740] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:22.920351 24440 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:22.836958 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.920876 24460 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:22.921506 22107 raft_consensus.cc:2393] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 7: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:22.701885 22077 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:22.881901 24204 raft_consensus.cc:2749] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:22.652104 24400 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.781147 21961 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:22.884807 24425 raft_consensus.cc:740] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:22.848587 22032 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:22.933396 22075 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:22.933543 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.935025 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.935645 22032 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:22.935926 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:22.795593 24453 consensus_queue.cc:1059] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20251023 02:05:22.825186 24292 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:22.857836 21925 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:22.943281 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 85559.1us
I20251023 02:05:22.945025 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.862206 22111 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251023 02:05:22.863381 22000 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.870231 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.873337 22102 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20251023 02:05:22.831758 24431 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.000s sys 0.020s
I20251023 02:05:22.808580 24436 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:22.863907 22616 raft_consensus.cc:2393] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:22.839049 24310 raft_consensus.cc:740] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:22.850064 24186 raft_consensus.cc:2749] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:22.859310 24416 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:22.859127 24495 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:22.895259 22035 raft_consensus.cc:2368] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6.
I20251023 02:05:22.898269 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.897249 22103 rpcz_store.cc:276] 1023 02:05:21.346222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.464358 (+118136us) service_pool.cc:225] Handling call
1023 02:05:22.828417 (+1364059us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54001ab3d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:22.828521 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":458105088}
I20251023 02:05:22.889494 24497 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:22.889170 24437 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:23.002048 24463 consensus_meta.cc:287] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.605s user 0.004s sys 0.005s
I20251023 02:05:23.002235 21959 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:23.002460 24463 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:23.003566 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:23.004508 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1571) took 1220 ms. Trace:
I20251023 02:05:23.004791 21959 rpcz_store.cc:276] 1023 02:05:21.783883 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.003249 (+1219366us) service_pool.cc:225] Handling call
1023 02:05:23.004456 (+ 1207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:22.905088 24481 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:22.905323 24405 raft_consensus.cc:3055] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:22.906283 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.906790 24290 raft_consensus.cc:697] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:22.907665 24144 raft_consensus.cc:3055] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:22.908237 24275 raft_consensus.cc:3055] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:22.910296 24323 raft_consensus.cc:3055] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:22.910507 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1233) took 1068 ms. Trace:
I20251023 02:05:22.912139 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1228) took 1072 ms. Trace:
I20251023 02:05:22.913990 22617 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:22.915131 24052 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:22.916611 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:22.916945 24467 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:22.872785 24003 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251023 02:05:22.922473 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 220647us
I20251023 02:05:23.055301 22077 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:22.839354 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:22.835160 24514 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.057958 22106 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:22.926951 22107 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:22.929977 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1230) took 1089 ms. Trace:
I20251023 02:05:22.845860 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.665264 24193 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.162s
I20251023 02:05:22.939406 21928 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:23.059087 21961 rpcz_store.cc:276] 1023 02:05:21.840408 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.618678 (+778270us) service_pool.cc:225] Handling call
1023 02:05:22.929923 (+311245us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.060272 22077 leader_election.cc:400] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6.
I20251023 02:05:22.940582 21954 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:22.945542 21925 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:23.060693 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 121471us
W20251023 02:05:23.061125 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 115691us
I20251023 02:05:23.061416 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:23.060740 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:22.946897 22111 raft_consensus.cc:1275] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251023 02:05:23.062140 21954 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:23.063455 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1570) took 1281 ms. Trace:
I20251023 02:05:23.063730 21954 rpcz_store.cc:276] 1023 02:05:21.782536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.061139 (+1278603us) service_pool.cc:225] Handling call
1023 02:05:23.063404 (+ 2265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:22.947265 22076 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:23.060688 22077 leader_election.cc:403] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20251023 02:05:23.064580 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 117401us
I20251023 02:05:23.066252 22076 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:22.947937 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 84622us
I20251023 02:05:23.067507 22000 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:22.948954 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:22.897047 24291 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.008s sys 0.023s
I20251023 02:05:22.844049 24295 consensus_queue.cc:260] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:23.072036 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:23.073287 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:22.964321 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:22.972564 21955 raft_consensus.cc:2393] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 4: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:05:22.980721 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:22.982136 22032 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:23.076218 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.885313 24339 consensus_queue.cc:1059] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20251023 02:05:22.991600 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:23.080490 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:22.993614 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:23.081056 21959 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:22.992789 21962 raft_consensus.cc:2410] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20251023 02:05:22.996867 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:22.942909 24547 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election won for term 6
W20251023 02:05:22.965727 24431 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.988s user 0.001s sys 0.020s
I20251023 02:05:22.942970 24292 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:23.000821 24497 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:23.005234 24463 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:23.114233 21956 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:22.957271 24483 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:23.080004 24484 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:23.016718 22028 raft_consensus.cc:2393] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:23.017607 21953 rpcz_store.cc:276] 1023 02:05:21.840094 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.039716 (+199622us) service_pool.cc:225] Handling call
1023 02:05:22.911944 (+872228us) spinlock_profiling.cc:243] Waited 45.5 ms on lock 0x7b540030e8d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:22.912075 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":104615808}
I20251023 02:05:23.017455 21958 rpcz_store.cc:276] 1023 02:05:21.842313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.862641 (+ 20328us) service_pool.cc:225] Handling call
1023 02:05:22.910325 (+1047684us) spinlock_profiling.cc:243] Waited 712 ms on lock 0x7b540030edd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:22.910453 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1637748992}
I20251023 02:05:23.051901 24436 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:22.997637 24503 consensus_meta.cc:287] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.690s user 0.007s sys 0.004s
W20251023 02:05:23.071833 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:23.072324 22000 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:23.116217 21959 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
W20251023 02:05:23.136747 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 64492.8us
I20251023 02:05:23.077162 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:22.931727 24440 raft_consensus.cc:2749] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:23.077756 22002 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:23.078610 21925 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:23.020943 24144 raft_consensus.cc:740] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:23.082027 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1575) took 1282 ms. Trace:
I20251023 02:05:23.113000 24350 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20251023 02:05:23.084892 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:23.016369 24405 raft_consensus.cc:740] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:23.072058 24291 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s user 0.009s sys 0.023s
I20251023 02:05:23.021023 24275 raft_consensus.cc:740] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:23.112032 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:23.071622 24403 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:23.072531 24525 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:23.055184 22623 raft_consensus.cc:1240] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:23.021093 24323 raft_consensus.cc:740] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:23.006615 22110 raft_consensus.cc:1240] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251023 02:05:22.999327 24416 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.073030 24466 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:22.962263 24470 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:23.073222 24368 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:23.085214 24467 raft_consensus.cc:2749] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:23.137871 22000 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:23.109666 24457 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:23.134931 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:23.139988 21928 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:23.141372 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 63671.6us
I20251023 02:05:23.105921 24431 raft_consensus.cc:697] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:23.144511 22031 raft_consensus.cc:2393] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:23.144522 21959 raft_consensus.cc:1240] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 7. Ops: []
I20251023 02:05:23.146112 22102 raft_consensus.cc:1275] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251023 02:05:23.146749 24453 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:23.134464 24310 consensus_queue.cc:260] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.116312 24294 raft_consensus.cc:3060] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 1 FOLLOWER]: Advancing to term 2
I20251023 02:05:23.133539 24437 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:23.080050 24543 consensus_queue.cc:1048] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20251023 02:05:23.152829 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:23.154086 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:23.142889 24484 raft_consensus.cc:697] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:23.156394 24440 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6.
I20251023 02:05:23.156962 24436 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:23.160598 24204 raft_consensus.cc:3055] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:23.164481 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1220) took 1346 ms. Trace:
I20251023 02:05:23.113688 24292 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
W20251023 02:05:23.168337 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 89825.5us
I20251023 02:05:23.174189 21962 rpcz_store.cc:276] 1023 02:05:21.799425 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.898038 (+1098613us) service_pool.cc:225] Handling call
1023 02:05:23.081960 (+183922us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.082041 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:23.156111 24507 consensus_meta.cc:287] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.280s user 0.003s sys 0.010s
I20251023 02:05:23.141044 24444 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:23.177173 22036 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:23.021428 24425 consensus_queue.cc:260] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:23.182319 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1456) took 1069 ms. Trace:
I20251023 02:05:23.223199 24389 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:23.254689 22030 rpcz_store.cc:276] 1023 02:05:22.112593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.084687 (+972094us) service_pool.cc:225] Handling call
1023 02:05:23.182265 (+ 97578us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.255110 21958 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:23.187624 21961 raft_consensus.cc:2410] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20251023 02:05:23.254994 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1204) took 4449 ms. Trace:
I20251023 02:05:23.153591 24524 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:23.185930 24544 log.cc:927] Time spent T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20251023 02:05:23.182870 24481 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:23.204202 22618 raft_consensus.cc:1240] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:23.209376 24144 consensus_queue.cc:260] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:23.144182 24503 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:23.215504 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:23.240758 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 102929us
I20251023 02:05:23.198940 24400 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:23.166467 24290 consensus_queue.cc:237] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.256009 21959 rpcz_store.cc:276] 1023 02:05:18.805557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.143620 (+4338063us) service_pool.cc:225] Handling call
1023 02:05:23.144231 (+ 611us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.254717 (+110486us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:23.254940 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.255910 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1239) took 1182 ms. Trace:
I20251023 02:05:23.257265 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1454) took 1145 ms. Trace:
I20251023 02:05:23.258634 22002 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:23.197201 24483 consensus_queue.cc:1059] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20251023 02:05:23.203157 24403 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.349752 22102 raft_consensus.cc:1240] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
W20251023 02:05:23.303293 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 47798.9us
W20251023 02:05:23.408732 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 348604us
I20251023 02:05:23.307418 22106 raft_consensus.cc:1240] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:23.260315 21953 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:23.395578 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:23.708266 22036 raft_consensus.cc:1275] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20251023 02:05:23.717188 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:23.725700 22029 rpcz_store.cc:276] 1023 02:05:22.112067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.081815 (+969748us) service_pool.cc:225] Handling call
1023 02:05:23.257205 (+175390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.379377 21958 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:23.726861 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:23.711745 21962 raft_consensus.cc:1240] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:23.726114 22000 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:23.407196 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 341050us
I20251023 02:05:23.727766 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1205) took 4918 ms. Trace:
I20251023 02:05:23.712276 21961 rpcz_store.cc:276] 1023 02:05:22.073575 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.062927 (+989352us) service_pool.cc:225] Handling call
1023 02:05:23.255858 (+192931us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.728541 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1449) took 3550 ms. Trace:
W20251023 02:05:23.711449 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:23.712908 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 454324us
W20251023 02:05:23.712328 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 572440us
I20251023 02:05:23.740968 21958 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20251023 02:05:23.767055 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:23.729373 21962 rpcz_store.cc:276] 1023 02:05:18.808881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.260048 (+4451167us) service_pool.cc:225] Handling call
1023 02:05:23.711577 (+451529us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.727514 (+ 15937us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:23.727713 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.758131 21925 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:23.731292 21961 raft_consensus.cc:1240] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:23.767904 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 172) took 4956 ms. Trace:
W20251023 02:05:23.768034 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:23.768334 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1548) took 4917 ms. Trace:
I20251023 02:05:23.768512 21925 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:23.768718 21958 rpcz_store.cc:276] 1023 02:05:18.811340 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.740212 (+4928872us) service_pool.cc:225] Handling call
1023 02:05:23.740574 (+ 362us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.767149 (+ 26575us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:23.767354 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.712399 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1635) took 1049 ms. Trace:
W20251023 02:05:23.348337 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 232538us
I20251023 02:05:23.765969 22036 rpcz_store.cc:276] 1023 02:05:20.178019 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.774297 (+1596278us) service_pool.cc:225] Handling call
1023 02:05:21.774667 (+ 370us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:22.595988 (+821321us) spinlock_profiling.cc:243] Waited 97.2 ms on lock 0x7b540031d6a0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa16fbf 00007f9c5fa8fb00 00007f9c5fa9137e 00007f9c5fa9b6d9 00007f9c5fa9bfa6 00007f9c5fa9892f 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2
1023 02:05:23.176589 (+580601us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b5800823dd0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5f9f4ffa 00007f9c5fa77097 00007f9c5fa8fb87 00007f9c5fa9137e 00007f9c5fa9b6d9 00007f9c5fa9bfa6 00007f9c5fa9892f 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34
1023 02:05:23.728247 (+551658us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:23.728488 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":995750528}
W20251023 02:05:23.772404 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:23.772886 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:23.724988 22028 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:23.774199 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1251) took 1243 ms. Trace:
W20251023 02:05:23.262542 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:23.774600 22028 rpcz_store.cc:276] 1023 02:05:22.530389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.134630 (+604241us) service_pool.cc:225] Handling call
1023 02:05:23.774131 (+639501us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.813067 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:23.776997 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1257) took 1144 ms. Trace:
I20251023 02:05:23.820878 22102 rpcz_store.cc:276] 1023 02:05:22.662883 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.259765 (+596882us) service_pool.cc:225] Handling call
1023 02:05:23.349499 (+ 89734us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.712092 (+362593us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:23.712347 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.813134 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:23.256762 21957 rpcz_store.cc:276] 1023 02:05:21.817919 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.998203 (+180284us) service_pool.cc:225] Handling call
1023 02:05:23.164264 (+1166061us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b540024ccd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:23.164406 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1250660608}
I20251023 02:05:23.768872 21961 rpcz_store.cc:276] 1023 02:05:18.851446 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.730810 (+4879364us) service_pool.cc:225] Handling call
1023 02:05:23.731105 (+ 295us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.768119 (+ 37014us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:23.768288 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.821867 22076 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:23.824642 22036 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:23.821501 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:23.826462 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:23.824383 22029 rpcz_store.cc:276] 1023 02:05:22.632118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.726618 (+1094500us) service_pool.cc:225] Handling call
1023 02:05:23.776932 (+ 50314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.829306 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:23.829900 21962 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:23.250015 24418 raft_consensus.cc:3055] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
W20251023 02:05:23.830415 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 61981us
W20251023 02:05:23.685340 24554 log.cc:927] Time spent T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.286s user 0.000s sys 0.000s
W20251023 02:05:23.709981 24550 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20251023 02:05:23.845849 21958 raft_consensus.cc:1240] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
W20251023 02:05:23.358807 24523 consensus_meta.cc:287] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.080s user 0.013s sys 0.000s
W20251023 02:05:23.865850 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:23.469694 24548 log.cc:927] Time spent T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20251023 02:05:23.732416 24543 raft_consensus.cc:2749] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:23.254482 24186 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251023 02:05:23.496115 24347 consensus_meta.cc:287] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.796s user 0.007s sys 0.004s
I20251023 02:05:23.335605 24425 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:23.681715 24449 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.143s user 0.010s sys 0.013s
I20251023 02:05:23.256644 24204 raft_consensus.cc:740] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:23.776584 24452 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20251023 02:05:23.301635 24470 raft_consensus.cc:697] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:23.301213 24437 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:23.670895 24389 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:23.730792 24467 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:23.358500 24514 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:23.712206 24525 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:23.702541 24416 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:23.305548 24440 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:23.305140 24310 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:23.305305 24436 raft_consensus.cc:697] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:23.776397 24537 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251023 02:05:23.836994 24552 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:23.837812 24418 raft_consensus.cc:740] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:23.845165 24368 raft_consensus.cc:2749] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:23.262303 24507 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:23.883776 24449 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.351s user 0.012s sys 0.023s
I20251023 02:05:23.253635 24405 consensus_queue.cc:260] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.334805 24323 consensus_queue.cc:260] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.916034 24449 raft_consensus.cc:3055] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:23.916429 24449 raft_consensus.cc:740] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:23.917163 24449 consensus_queue.cc:260] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.907229 24525 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.869515 24523 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.304855 24484 consensus_queue.cc:237] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.907058 24467 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:23.915194 24507 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:23.918116 24449 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:23.302726 24431 consensus_queue.cc:237] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.924257 24467 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:23.913028 24552 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:23.923698 24470 consensus_queue.cc:237] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.933355 24436 consensus_queue.cc:237] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:23.934605 24497 consensus_meta.cc:287] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.802s user 0.009s sys 0.000s
I20251023 02:05:23.935037 24497 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.910307 24440 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.937754 24507 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:23.938266 24507 raft_consensus.cc:697] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:23.939072 24507 consensus_queue.cc:237] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.940816 24523 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:23.941493 24523 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:23.942072 24523 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.948436 24552 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:23.937237 24469 consensus_meta.cc:287] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.231s user 0.007s sys 0.003s
I20251023 02:05:23.927773 24503 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:23.949354 24469 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:23.944422 24523 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:23.846565 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1550) took 4994 ms. Trace:
W20251023 02:05:23.977533 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:23.844354 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1452) took 1732 ms. Trace:
I20251023 02:05:23.980186 22028 rpcz_store.cc:276] 1023 02:05:22.111516 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.826185 (+1714669us) service_pool.cc:225] Handling call
1023 02:05:23.844280 (+ 18095us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.865149 21961 raft_consensus.cc:1240] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 1. Current term is 2. Ops: []
I20251023 02:05:23.983786 22028 raft_consensus.cc:1275] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 2. (index mismatch)
I20251023 02:05:23.984303 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1551) took 5131 ms. Trace:
I20251023 02:05:23.984742 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1460) took 1871 ms. Trace:
I20251023 02:05:23.855002 21962 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:24.015477 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:23.845952 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1461) took 1731 ms. Trace:
I20251023 02:05:24.034509 22030 rpcz_store.cc:276] 1023 02:05:22.114184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.726704 (+1612520us) service_pool.cc:225] Handling call
1023 02:05:23.845895 (+119191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.036571 22030 raft_consensus.cc:1240] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:23.863380 22036 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:23.874567 22102 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:24.038219 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1473) took 1155 ms. Trace:
I20251023 02:05:24.038833 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1636) took 1375 ms. Trace:
I20251023 02:05:24.038753 22036 rpcz_store.cc:276] 1023 02:05:22.883160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.772292 (+889132us) service_pool.cc:225] Handling call
1023 02:05:24.037962 (+265670us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:23.884346 22105 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.822s user 0.038s sys 0.034s
I20251023 02:05:23.985201 22028 rpcz_store.cc:276] 1023 02:05:22.113881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.981940 (+1868059us) service_pool.cc:225] Handling call
1023 02:05:23.982874 (+ 934us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.984366 (+ 1492us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:23.984615 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:23.776799 24052 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:23.884593 24069 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.041028 22036 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251023 02:05:23.920357 24418 consensus_queue.cc:260] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.041658 24069 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:23.876065 24186 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:24.043385 24418 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:23.864123 24291 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.029s
I20251023 02:05:24.043852 24497 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:24.044258 22033 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:23.260038 24275 consensus_queue.cc:260] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.842193 24144 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:23.908938 24204 consensus_queue.cc:260] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.991695 24469 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:23.334838 24346 consensus_queue.cc:1175] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" responder_term: 7 status { last_received { term: 6 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251023 02:05:23.900537 24389 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:23.976701 21958 rpcz_store.cc:276] 1023 02:05:18.852194 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.770167 (+4917973us) service_pool.cc:225] Handling call
1023 02:05:23.845683 (+ 75516us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.846329 (+ 646us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:23.846513 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.009351 21960 raft_consensus.cc:2393] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:24.010370 22104 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251023 02:05:23.984771 21961 rpcz_store.cc:276] 1023 02:05:18.852485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.825284 (+4972799us) service_pool.cc:225] Handling call
1023 02:05:23.864936 (+ 39652us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.983704 (+118768us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:23.984091 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.023862 21962 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:24.028546 24403 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.043460 24186 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:23.990860 24467 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:24.039397 22102 rpcz_store.cc:276] 1023 02:05:22.663234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.835041 (+1171807us) service_pool.cc:225] Handling call
1023 02:05:23.874337 (+ 39296us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.038168 (+163831us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.038559 (+ 391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.046061 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1463) took 1922 ms. Trace:
I20251023 02:05:24.117723 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1465) took 1999 ms. Trace:
I20251023 02:05:24.154894 22036 rpcz_store.cc:276] 1023 02:05:22.115330 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.039813 (+1924483us) service_pool.cc:225] Handling call
1023 02:05:24.040384 (+ 571us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.041539 (+ 1155us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.114805 (+ 73266us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.045960 24275 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:24.154148 22030 rpcz_store.cc:276] 1023 02:05:22.114816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.035598 (+1920782us) service_pool.cc:225] Handling call
1023 02:05:24.036135 (+ 537us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.037035 (+ 900us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.037277 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.159252 24457 raft_consensus.cc:697] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:24.161478 24457 consensus_queue.cc:237] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:24.174955 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 298688us
I20251023 02:05:24.209674 24339 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.209705 24052 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.210886 24339 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:24.213039 24052 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.213222 24339 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:24.209775 24347 raft_consensus.cc:2764] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20251023 02:05:24.205700 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1457) took 2092 ms. Trace:
I20251023 02:05:24.219381 24537 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:24.224726 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:24.244459 24469 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.226467 24552 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:24.288143 24552 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:24.288307 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 561510us
I20251023 02:05:24.290861 24552 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.289144 24469 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:24.226307 21953 consensus_meta.cc:287] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.511s user 0.013s sys 0.002s
I20251023 02:05:24.240348 24425 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:24.215199 24389 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.239163 22619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 359) took 2256 ms. Trace:
I20251023 02:05:24.196941 24449 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.251497 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1475) took 1355 ms. Trace:
W20251023 02:05:24.253131 22035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1225) took 2166 ms (client timeout 1720 ms). Trace:
I20251023 02:05:24.242676 24568 consensus_queue.cc:1048] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251023 02:05:24.247184 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 424386us
I20251023 02:05:24.222826 22102 raft_consensus.cc:1240] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:24.215219 24204 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
W20251023 02:05:24.209728 22105 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.075s user 0.039s sys 0.035s
I20251023 02:05:24.282056 22029 rpcz_store.cc:276] 1023 02:05:22.112853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.827646 (+1714793us) service_pool.cc:225] Handling call
1023 02:05:23.827995 (+ 349us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.204858 (+376863us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:24.204927 (+ 69us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:24.205002 (+ 75us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:24.205080 (+ 78us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.205207 (+ 127us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.205453 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16725,"raft.queue_time_us":85003,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":3150,"thread_start_us":354445,"threads_started":1}
I20251023 02:05:24.249997 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:05:24.199841 24403 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.308094 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1637) took 1641 ms. Trace:
W20251023 02:05:24.309270 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 486152us
W20251023 02:05:24.303380 22035 rpcz_store.cc:269] 1023 02:05:22.086219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.088514 (+ 2295us) service_pool.cc:225] Handling call
1023 02:05:24.252518 (+2164004us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b540026f054. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:24.252778 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1766227200}
I20251023 02:05:24.311187 22102 rpcz_store.cc:276] 1023 02:05:22.663496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.221813 (+1558317us) service_pool.cc:225] Handling call
1023 02:05:24.222466 (+ 653us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.304874 (+ 82408us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.305105 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.252316 21858 catalog_manager.cc:5649] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: config changed from index 2 to 4, term changed from 4 to 6, leader changed from 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1), VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2), VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3), VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 6 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:24.310834 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:24.303303 22031 rpcz_store.cc:276] 1023 02:05:22.895882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.946126 (+ 50244us) service_pool.cc:225] Handling call
1023 02:05:24.251024 (+1304898us) spinlock_profiling.cc:243] Waited 59.5 ms on lock 0x7b540026f554. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:24.251250 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":136770432}
I20251023 02:05:24.312748 21953 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
W20251023 02:05:24.318998 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 532316us
I20251023 02:05:24.322505 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1503) took 1824 ms. Trace:
I20251023 02:05:24.324292 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 252) took 1853 ms. Trace:
I20251023 02:05:24.322948 22111 rpcz_store.cc:276] 1023 02:05:22.498468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.063633 (+565165us) service_pool.cc:225] Handling call
1023 02:05:23.063926 (+ 293us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.064114 (+ 188us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:23.407884 (+343770us) log.cc:844] Serialized 14 byte log entry
1023 02:05:24.310221 (+902337us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:24.321794 (+ 11573us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:24.321969 (+ 175us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.322082 (+ 113us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.322299 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":824073,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":49119,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":88,"spinlock_wait_cycles":200875520,"thread_start_us":1089572,"threads_started":3,"wal-append.queue_time_us":798483}
I20251023 02:05:24.324666 22106 rpcz_store.cc:276] 1023 02:05:22.470795 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.058842 (+588047us) service_pool.cc:225] Handling call
1023 02:05:23.111319 (+ 52477us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.713609 (+602290us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.323862 (+610253us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b540013f454. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:24.324134 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":450457344}
I20251023 02:05:24.326143 24403 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:24.326869 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 45989.2us
W20251023 02:05:24.327311 22616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34806 (request call id 356) took 2973 ms (client timeout 2469 ms). Trace:
I20251023 02:05:24.327816 22002 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:24.327786 22616 rpcz_store.cc:269] 1023 02:05:21.353921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.361716 (+ 7795us) service_pool.cc:225] Handling call
1023 02:05:24.326863 (+2965147us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400181854. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:24.327084 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2717053312}
I20251023 02:05:24.328823 24403 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.329902 24467 raft_consensus.cc:2804] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:24.330271 22623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 372) took 1798 ms. Trace:
I20251023 02:05:24.330516 24467 raft_consensus.cc:697] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:24.330773 22623 rpcz_store.cc:276] 1023 02:05:22.531520 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.531733 (+ 213us) service_pool.cc:225] Handling call
1023 02:05:22.543714 (+ 11981us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.203112 (+659398us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.329777 (+1126665us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b54005ce9d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:24.330082 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1059103104}
I20251023 02:05:24.331874 24467 consensus_queue.cc:237] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.332799 22622 raft_consensus.cc:1240] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
W20251023 02:05:24.315780 22107 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.830s user 0.013s sys 0.070s
W20251023 02:05:24.334129 22107 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.992s user 0.014s sys 0.070s
I20251023 02:05:24.335326 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1505) took 1771 ms. Trace:
W20251023 02:05:24.335419 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:24.335752 22107 rpcz_store.cc:276] 1023 02:05:22.564310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.084663 (+520353us) service_pool.cc:225] Handling call
1023 02:05:23.084939 (+ 276us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.085146 (+ 207us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:05:23.341898 (+256752us) log.cc:844] Serialized 14 byte log entry
1023 02:05:24.334730 (+992832us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:24.334832 (+ 102us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:24.334964 (+ 132us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.335091 (+ 127us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.335266 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22160,"prepare.queue_time_us":809245,"prepare.run_cpu_time_us":1305,"prepare.run_wall_time_us":5789,"thread_start_us":1200010,"threads_started":2,"wal-append.queue_time_us":972869}
W20251023 02:05:24.337111 22077 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.712s (SENT) [suppressed 31 similar messages]
I20251023 02:05:24.331396 22618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 365) took 1251 ms. Trace:
I20251023 02:05:24.325384 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.322743 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1627) took 2229 ms. Trace:
I20251023 02:05:24.309921 24339 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.344592 24481 raft_consensus.cc:697] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:24.345010 24449 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.345753 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 251) took 1943 ms. Trace:
I20251023 02:05:24.346159 24481 consensus_queue.cc:237] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.346212 22105 rpcz_store.cc:276] 1023 02:05:22.402748 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.012595 (+609847us) service_pool.cc:225] Handling call
1023 02:05:23.012937 (+ 342us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:23.059318 (+ 46381us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:23.060087 (+ 769us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:23.062261 (+ 2174us) log.cc:844] Serialized 26 byte log entry
1023 02:05:24.310853 (+1248592us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:24.315257 (+ 4404us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.338352 (+ 23095us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:24.338400 (+ 48us) raft_consensus.cc:1685] finished
1023 02:05:24.338412 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.345692 (+ 7280us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":37048,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":1242902,"threads_started":1,"wal-append.queue_time_us":1171360}
I20251023 02:05:24.350788 21954 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:24.338403 22618 rpcz_store.cc:276] 1023 02:05:23.079535 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.079794 (+ 259us) service_pool.cc:225] Handling call
1023 02:05:23.109469 (+ 29675us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:23.728387 (+618918us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.330926 (+602539us) spinlock_profiling.cc:243] Waited 94.5 ms on lock 0x7b5400171e54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:24.331206 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":217365760}
W20251023 02:05:24.353716 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:24.343731 22110 rpcz_store.cc:276] 1023 02:05:22.093257 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.898061 (+804804us) service_pool.cc:225] Handling call
1023 02:05:22.898331 (+ 270us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:23.200407 (+302076us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.322265 (+1121858us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54000118d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:24.322564 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":248304512}
W20251023 02:05:24.354437 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:24.355855 22106 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20251023 02:05:24.356587 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 255) took 1432 ms. Trace:
I20251023 02:05:24.356946 22106 rpcz_store.cc:276] 1023 02:05:22.924580 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.354764 (+1430184us) service_pool.cc:225] Handling call
1023 02:05:24.355308 (+ 544us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.356327 (+ 1019us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.356519 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.301270 22619 rpcz_store.cc:276] 1023 02:05:21.982209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.986558 (+ 4349us) service_pool.cc:225] Handling call
1023 02:05:21.986885 (+ 327us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:22.273094 (+286209us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:22.276404 (+ 3310us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:22.336735 (+ 60331us) log.cc:844] Serialized 26 byte log entry
1023 02:05:22.746976 (+410241us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:23.112702 (+365726us) log.cc:844] Serialized 14 byte log entry
1023 02:05:23.116455 (+ 3753us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.238347 (+1121892us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b54005fdf54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:24.238411 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:24.238520 (+ 109us) raft_consensus.cc:1685] finished
1023 02:05:24.238641 (+ 121us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.238914 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":538602,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":43337,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":27423,"spinlock_wait_cycles":38152320,"thread_start_us":735740,"threads_started":3,"wal-append.queue_time_us":542929}
W20251023 02:05:24.353309 22077 leader_election.cc:336] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 2.075s (SENT)
I20251023 02:05:24.354251 22587 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:24.370663 22103 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:24.396016 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1244) took 2184 ms. Trace:
W20251023 02:05:24.371253 24292 consensus_meta.cc:287] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.065s user 0.007s sys 0.009s
I20251023 02:05:24.396488 21953 rpcz_store.cc:276] 1023 02:05:22.187109 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.133765 (+946656us) service_pool.cc:225] Handling call
1023 02:05:24.362705 (+1228940us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.348716 24449 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.443686 22102 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20251023 02:05:24.415520 21962 raft_consensus.cc:2410] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20251023 02:05:24.426019 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 133882us
I20251023 02:05:24.420251 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 248) took 2263 ms. Trace:
I20251023 02:05:24.421965 21928 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:24.449518 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1638) took 1785 ms. Trace:
I20251023 02:05:24.450019 22102 rpcz_store.cc:276] 1023 02:05:22.663733 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.343991 (+1680258us) service_pool.cc:225] Handling call
1023 02:05:24.443382 (+ 99391us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:24.449183 (+ 5801us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.449456 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.452217 22111 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:24.399484 22106 raft_consensus.cc:1275] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251023 02:05:24.421600 24444 raft_consensus.cc:3055] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:24.399559 24469 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.454953 24444 raft_consensus.cc:740] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:24.423139 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 152508us
I20251023 02:05:24.456614 21928 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:24.456333 24444 consensus_queue.cc:260] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:24.458003 24469 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:24.458611 22107 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251023 02:05:24.458787 24469 raft_consensus.cc:2749] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251023 02:05:24.451603 22076 connection.cc:232] RPC call timeout handler was delayed by -1.0975375919998669s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16000000000000003s for a response
W20251023 02:05:24.466233 22077 leader_election.cc:336] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 2.133s (SENT)
I20251023 02:05:24.468753 22029 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: [3.3-6.5]
I20251023 02:05:24.447075 22028 rpcz_store.cc:276] 1023 02:05:22.157308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.154260 (+1996952us) service_pool.cc:225] Handling call
1023 02:05:24.223236 (+ 68976us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:24.223934 (+ 698us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:24.223973 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:24.224896 (+ 923us) log.cc:844] Serialized 52 byte log entry
1023 02:05:24.305865 (+ 80969us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:24.305936 (+ 71us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.306010 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:24.420023 (+114013us) raft_consensus.cc:1685] finished
1023 02:05:24.420049 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.420194 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10400,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":245,"spinlock_wait_cycles":494976,"thread_start_us":80498,"threads_started":1,"wal-append.queue_time_us":192417}
I20251023 02:05:24.472606 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1468) took 2115 ms. Trace:
I20251023 02:05:24.472967 22029 rpcz_store.cc:276] 1023 02:05:22.353769 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.339633 (+1985864us) service_pool.cc:225] Handling call
1023 02:05:24.459715 (+120082us) raft_consensus.cc:1471] Updating replica for 3 ops
1023 02:05:24.469249 (+ 9534us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.469477 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":442624}
I20251023 02:05:24.466744 22077 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:24.476841 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:24.478121 22031 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:24.479063 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1469) took 2122 ms. Trace:
I20251023 02:05:24.479523 22031 rpcz_store.cc:276] 1023 02:05:22.356315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.343531 (+1987216us) service_pool.cc:225] Handling call
1023 02:05:24.464252 (+120721us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.478694 (+ 14442us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.478988 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.482056 24574 consensus_queue.cc:1059] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:05:24.482048 24310 consensus_queue.cc:1059] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:05:24.484436 24433 raft_consensus.cc:3055] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:24.484053 24310 consensus_queue.cc:1048] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251023 02:05:24.484838 24433 raft_consensus.cc:740] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:24.485854 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:24.486099 24204 raft_consensus.cc:3055] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:24.486778 24204 raft_consensus.cc:740] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:24.488503 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39634 (request call id 853) took 2353 ms. Trace:
I20251023 02:05:24.488199 24204 consensus_queue.cc:260] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:24.489250 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:24.491878 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:24.492386 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:24.492739 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:24.493983 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:24.494104 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 140197us
I20251023 02:05:24.484149 24574 consensus_queue.cc:1059] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:05:24.488906 22110 rpcz_store.cc:276] 1023 02:05:22.158839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.354572 (+2195733us) service_pool.cc:225] Handling call
1023 02:05:24.488411 (+133839us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.377882 24425 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.468737 24444 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:24.498979 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:24.497318 22031 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:24.423079 24339 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.500507 24291 consensus_queue.cc:1059] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20251023 02:05:24.405090 24537 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:24.506145 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:24.506726 22035 raft_consensus.cc:1240] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:24.507436 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:24.460757 24292 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.508701 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:24.528234 22616 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:24.528338 24575 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.528739 24465 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.510895 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:24.511356 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:24.498255 24552 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251023 02:05:24.514490 22620 raft_consensus.cc:1240] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:24.514554 24468 raft_consensus.cc:3055] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:24.515163 24402 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.515628 24583 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.516405 22002 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:24.516502 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1260) took 1883 ms. Trace:
I20251023 02:05:24.556317 22029 rpcz_store.cc:276] 1023 02:05:22.632955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.498802 (+1865847us) service_pool.cc:225] Handling call
1023 02:05:24.515671 (+ 16869us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.516057 (+ 386us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:24.516085 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:24.516189 (+ 104us) raft_consensus.cc:1645] Marking committed up to 8
1023 02:05:24.516231 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.516297 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.516429 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:24.516916 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:24.517935 24579 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251023 02:05:24.518095 24294 consensus_meta.cc:287] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.257s user 0.018s sys 0.001s
W20251023 02:05:24.518102 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 54967.4us
I20251023 02:05:24.520176 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1508) took 1898 ms. Trace:
I20251023 02:05:24.522011 24577 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.522888 22028 raft_consensus.cc:1240] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:24.523471 24433 consensus_queue.cc:260] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:24.521169 22076 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:24.526517 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 249) took 1824 ms. Trace:
I20251023 02:05:24.571034 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1259) took 1938 ms. Trace:
I20251023 02:05:24.571230 22031 rpcz_store.cc:276] 1023 02:05:22.701859 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.480681 (+1778822us) service_pool.cc:225] Handling call
1023 02:05:24.481284 (+ 603us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.524876 (+ 43592us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.526447 (+ 1571us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.571406 22028 rpcz_store.cc:276] 1023 02:05:22.632699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.497120 (+1864421us) service_pool.cc:225] Handling call
1023 02:05:24.522634 (+ 25514us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.570780 (+ 48146us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.570983 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.529461 22035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1470) took 1989 ms. Trace:
W20251023 02:05:24.535104 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 78624.8us
I20251023 02:05:24.537102 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.537412 21954 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:24.536018 24465 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.545189 24402 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.580211 22029 raft_consensus.cc:1240] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:24.580844 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.581128 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 251) took 1867 ms. Trace:
I20251023 02:05:24.581751 22111 raft_consensus.cc:2393] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:24.581861 24402 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.546085 24537 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:24.538285 24468 raft_consensus.cc:740] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:24.538213 22102 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:24.551373 22623 raft_consensus.cc:2393] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 4: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:24.584302 24465 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.583915 24468 consensus_queue.cc:260] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.582861 22109 raft_consensus.cc:1240] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9. Ops: []
I20251023 02:05:24.553629 24204 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:24.534085 22000 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:24.585469 24468 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:24.580161 24433 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:24.586508 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 104942us
I20251023 02:05:24.564056 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:24.587860 21962 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:24.525274 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.572373 22035 rpcz_store.cc:276] 1023 02:05:22.539792 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.401499 (+1861707us) service_pool.cc:225] Handling call
1023 02:05:24.506503 (+105004us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.527508 (+ 21005us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.529410 (+ 1902us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:24.572811 21954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36488 (request call id 193) took 1509 ms (client timeout 1996 ms). Trace:
I20251023 02:05:24.567557 24576 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:24.569998 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:24.564023 22103 rpcz_store.cc:276] 1023 02:05:22.621317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.081649 (+460332us) service_pool.cc:225] Handling call
1023 02:05:23.114543 (+ 32894us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.371226 (+1256683us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.519919 (+148693us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54000fbc54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:24.520115 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2879169280}
W20251023 02:05:24.573417 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 57274.8us
I20251023 02:05:24.581712 22029 rpcz_store.cc:276] 1023 02:05:22.714039 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.568148 (+1854109us) service_pool.cc:225] Handling call
1023 02:05:24.579876 (+ 11728us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.580815 (+ 939us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.581041 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.589749 21953 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:24.582851 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:24.588714 21954 rpcz_store.cc:269] 1023 02:05:23.063062 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.064295 (+ 1233us) service_pool.cc:225] Handling call
1023 02:05:24.249415 (+1185120us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b5800002a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:05:24.572759 (+323344us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6360448}
I20251023 02:05:24.599027 22111 raft_consensus.cc:2393] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:24.529067 24575 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:24.600217 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:24.602591 21962 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:24.592633 24465 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:24.598769 24576 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:24.601773 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 106029us
I20251023 02:05:24.604789 22106 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:24.621474 22110 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:24.562397 24186 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.604801 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.624367 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:24.623204 24537 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.629349 24457 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:24.621043 24310 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
W20251023 02:05:24.630209 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:24.632475 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 254) took 1616 ms. Trace:
I20251023 02:05:24.629768 24481 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:24.629634 24537 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.641238 24481 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.642973 24537 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.643296 24481 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.629634 22589 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:24.648139 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 46592.2us
I20251023 02:05:24.632270 24457 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:24.654330 21962 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:24.654404 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.594060 24583 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.656194 22028 raft_consensus.cc:1240] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:24.659075 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:24.645709 22616 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:24.658972 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1466) took 2475 ms. Trace:
I20251023 02:05:24.659958 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.650389 24310 raft_consensus.cc:697] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:24.618938 24577 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.637455 24469 consensus_queue.cc:1048] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20251023 02:05:24.753834 21954 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:24.661136 22111 raft_consensus.cc:1240] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: [6.2-6.2]
W20251023 02:05:24.691552 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 91034.4us
W20251023 02:05:24.660900 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 139979us
I20251023 02:05:24.743187 22110 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:24.685784 22106 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:24.685895 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 256) took 1746 ms. Trace:
W20251023 02:05:24.764824 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:24.744803 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:24.661430 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1479) took 1731 ms. Trace:
I20251023 02:05:24.784631 22028 rpcz_store.cc:276] 1023 02:05:22.930349 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.593557 (+1663208us) service_pool.cc:225] Handling call
1023 02:05:24.635521 (+ 41964us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.661187 (+ 25666us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.661375 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.756402 22102 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:24.787483 21953 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:24.657475 24457 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:24.730975 21926 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:24.790617 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 48189.2us
I20251023 02:05:24.659950 24425 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:24.685909 24551 log.cc:927] Time spent T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 1.482s user 0.002s sys 0.009s
W20251023 02:05:24.687268 24588 log.cc:927] Time spent T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20251023 02:05:24.796926 22034 raft_consensus.cc:2468] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2.
I20251023 02:05:24.752812 24186 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:24.753159 24453 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.021s sys 0.043s
I20251023 02:05:24.799744 22589 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
I20251023 02:05:24.782155 22105 rpcz_store.cc:276] 1023 02:05:22.939350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.348090 (+1408740us) service_pool.cc:225] Handling call
1023 02:05:24.348613 (+ 523us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:24.349360 (+ 747us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:24.349392 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:24.424722 (+ 75330us) log.cc:844] Serialized 52 byte log entry
1023 02:05:24.468794 (+ 44072us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:24.468836 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.468934 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:24.685639 (+216705us) raft_consensus.cc:1685] finished
1023 02:05:24.685668 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.685818 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":125,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":113,"thread_start_us":42982,"threads_started":1,"wal-append.queue_time_us":173214}
I20251023 02:05:24.682087 24339 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:24.800352 24482 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:24.804450 24546 log.cc:927] Time spent T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 1.661s user 0.000s sys 0.003s
I20251023 02:05:24.804675 24575 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:24.804855 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 146131us
I20251023 02:05:24.656863 24294 consensus_queue.cc:1059] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.142s
I20251023 02:05:24.686462 24576 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:24.690479 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:24.806123 24600 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election won for term 4
W20251023 02:05:24.807768 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:24.686462 24583 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:24.808540 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 175320us
I20251023 02:05:24.794191 22036 rpcz_store.cc:276] 1023 02:05:22.184033 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.173630 (+1989597us) service_pool.cc:225] Handling call
1023 02:05:24.293628 (+119998us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.321980 (+ 28352us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:24.459703 (+137723us) log.cc:844] Serialized 14 byte log entry
1023 02:05:24.658562 (+198859us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:24.658613 (+ 51us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:24.658668 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.658799 (+ 131us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.658921 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8204,"prepare.queue_time_us":195022,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":11990,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":18680,"thread_start_us":287608,"threads_started":2,"wal-append.queue_time_us":216626}
W20251023 02:05:24.811833 21908 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet fb1e5f57f70948bcb887f55cb3702b0e, TS 818907d63f1f4d84ba267ebe6ee837e0 -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.690654 24465 consensus_queue.cc:1048] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.124s
W20251023 02:05:24.820869 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet fb1e5f57f70948bcb887f55cb3702b0e, TS 818907d63f1f4d84ba267ebe6ee837e0 -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.807693 22103 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:24.812407 22102 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20251023 02:05:24.806607 24453 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.938s user 0.021s sys 0.051s
I20251023 02:05:24.812799 24310 consensus_queue.cc:237] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.813427 24144 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251023 02:05:24.813898 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 83035.8us
I20251023 02:05:24.814435 22035 raft_consensus.cc:1275] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20251023 02:05:24.800698 24292 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:24.815778 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.816190 21962 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:24.827579 21926 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
W20251023 02:05:24.827945 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:24.831286 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:24.831903 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1561) took 5578 ms. Trace:
I20251023 02:05:24.830080 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.827337 22035 pending_rounds.cc:85] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0: Aborting all ops after (but not including) 3
I20251023 02:05:24.832746 22076 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:24.832413 21962 rpcz_store.cc:276] 1023 02:05:19.253899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.743332 (+5489433us) service_pool.cc:225] Handling call
1023 02:05:24.801720 (+ 58388us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.831672 (+ 29952us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.831848 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.832743 22035 pending_rounds.cc:107] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20251023 02:05:24.836606 22035 raft_consensus.cc:2889] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251023 02:05:24.838964 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:24.839542 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:24.840140 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:24.841074 24552 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:24.841188 24193 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:24.841377 22110 raft_consensus.cc:2393] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:24.841660 24552 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:24.842144 24552 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:24.842478 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.845752 21954 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:24.846395 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1564) took 4690 ms. Trace:
I20251023 02:05:24.846850 24577 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:24.846760 21954 rpcz_store.cc:276] 1023 02:05:20.155638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.801468 (+4645830us) service_pool.cc:225] Handling call
1023 02:05:24.825240 (+ 23772us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.846125 (+ 20885us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.846319 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.847553 24193 raft_consensus.cc:697] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:24.816195 24482 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:24.849014 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 183) took 4634 ms. Trace:
I20251023 02:05:24.848635 24193 consensus_queue.cc:237] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:24.824895 24453 raft_consensus.cc:697] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:24.850924 24453 consensus_queue.cc:237] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:24.851323 24578 raft_consensus.cc:3055] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:24.851775 24578 raft_consensus.cc:740] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:24.854004 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.854975 22103 raft_consensus.cc:2410] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 7 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251023 02:05:24.638945 22029 rpcz_store.cc:276] 1023 02:05:23.016375 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.596322 (+1579947us) service_pool.cc:225] Handling call
1023 02:05:24.599095 (+ 2773us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.627199 (+ 28104us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:24.627244 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:24.632118 (+ 4874us) raft_consensus.cc:1645] Marking committed up to 7
1023 02:05:24.632176 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.632261 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.632409 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21576,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":7773}
I20251023 02:05:24.855840 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:24.856170 22103 raft_consensus.cc:1275] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251023 02:05:24.859779 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:24.866508 22106 raft_consensus.cc:1275] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251023 02:05:24.866070 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 174) took 6009 ms. Trace:
I20251023 02:05:24.871768 22622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 367) took 3381 ms. Trace:
I20251023 02:05:24.866291 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:24.873631 22036 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
W20251023 02:05:24.874068 22034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55770 (request call id 246) took 3085 ms (client timeout 2158 ms). Trace:
W20251023 02:05:24.874544 22034 rpcz_store.cc:269] 1023 02:05:21.789046 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.805673 (+ 16627us) service_pool.cc:225] Handling call
1023 02:05:24.873813 (+3068140us) spinlock_profiling.cc:243] Waited 2.64 s on lock 0x7b54003aa054. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:24.873990 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6063490432}
I20251023 02:05:24.875016 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:24.868944 21957 rpcz_store.cc:276] 1023 02:05:18.855988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.825512 (+4969524us) service_pool.cc:225] Handling call
1023 02:05:23.825859 (+ 347us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:24.177096 (+351237us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:24.177159 (+ 63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:24.316871 (+139712us) log.cc:844] Serialized 26 byte log entry
1023 02:05:24.461306 (+144435us) raft_consensus.cc:1645] Marking committed up to 8
1023 02:05:24.834292 (+372986us) log.cc:844] Serialized 14 byte log entry
1023 02:05:24.863047 (+ 28755us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.863520 (+ 473us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:24.863972 (+ 452us) raft_consensus.cc:1685] finished
1023 02:05:24.863992 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.865110 (+ 1118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43604,"raft.queue_time_us":76803,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":3825,"thread_start_us":741980,"threads_started":3,"wal-append.queue_time_us":211424}
I20251023 02:05:24.877136 21961 raft_consensus.cc:1240] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:24.893766 21954 rpcz_store.cc:276] 1023 02:05:20.215452 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.847833 (+4632381us) service_pool.cc:225] Handling call
1023 02:05:24.848237 (+ 404us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.848676 (+ 439us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:24.848704 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:24.848761 (+ 57us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:24.848789 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.848850 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.848952 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1741,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73}
I20251023 02:05:24.878794 22036 raft_consensus.cc:1240] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20251023 02:05:24.988608 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 186634us
I20251023 02:05:24.992585 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:24.872241 22622 rpcz_store.cc:276] 1023 02:05:21.490408 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.695224 (+204816us) service_pool.cc:225] Handling call
1023 02:05:24.332377 (+2637153us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.333258 (+ 881us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.871544 (+538286us) spinlock_profiling.cc:243] Waited 2.64 s on lock 0x7b5400181850. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:24.871708 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6064442624}
I20251023 02:05:24.904814 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:24.933060 22034 raft_consensus.cc:1275] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251023 02:05:24.923974 22104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1618) took 2885 ms. Trace:
W20251023 02:05:24.914518 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 78597.6us
I20251023 02:05:24.955928 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:24.998258 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 257) took 1617 ms. Trace:
I20251023 02:05:24.999214 21962 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:24.999051 22034 rpcz_store.cc:276] 1023 02:05:23.380758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.875482 (+1494724us) service_pool.cc:225] Handling call
1023 02:05:24.903941 (+ 28459us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.997048 (+ 93107us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.998090 (+ 1042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:24.953799 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:24.993264 22076 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:25.001044 24576 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:24.924237 22105 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:24.996140 22029 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
W20251023 02:05:24.904553 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 70569.3us
I20251023 02:05:24.939287 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 256) took 1771 ms. Trace:
I20251023 02:05:24.996322 22104 rpcz_store.cc:276] 1023 02:05:22.024264 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.586248 (+561984us) service_pool.cc:225] Handling call
1023 02:05:22.586624 (+ 376us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:24.177835 (+1591211us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.879149 (+701314us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54001aaed4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:24.910064 (+ 30915us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3272738176}
I20251023 02:05:25.005524 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1552) took 6060 ms. Trace:
I20251023 02:05:25.015933 21961 rpcz_store.cc:276] 1023 02:05:18.942014 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.876488 (+5934474us) service_pool.cc:225] Handling call
1023 02:05:24.876893 (+ 405us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.996614 (+119721us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.001876 (+ 5262us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:25.010929 22033 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1444) took 4890 ms (client timeout 1849 ms). Trace:
I20251023 02:05:25.017812 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1467) took 2757 ms. Trace:
W20251023 02:05:25.010149 21955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1232) took 3168 ms (client timeout 2093 ms). Trace:
I20251023 02:05:25.010411 21953 raft_consensus.cc:1240] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:25.005999 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:25.007754 21958 raft_consensus.cc:1240] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:24.938788 24577 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:25.005115 22028 rpcz_store.cc:276] 1023 02:05:23.167503 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.816096 (+1648593us) service_pool.cc:225] Handling call
1023 02:05:24.831355 (+ 15259us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.914247 (+ 82892us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:24.914278 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:24.938984 (+ 24706us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:24.939023 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:24.939087 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:24.939222 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54591,"raft.queue_time_us":106185,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":30328}
I20251023 02:05:25.005466 24295 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:25.021665 24467 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.022025 22037 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251023 02:05:25.001477 24576 raft_consensus.cc:697] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
W20251023 02:05:25.017762 22033 rpcz_store.cc:269] 1023 02:05:20.120543 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:20.123740 (+ 3197us) service_pool.cc:225] Handling call
1023 02:05:25.010691 (+4886951us) spinlock_profiling.cc:243] Waited 3.89 s on lock 0x7b5400280d54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.010838 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8958192512}
I20251023 02:05:25.024425 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1553) took 6078 ms. Trace:
W20251023 02:05:25.018970 21955 rpcz_store.cc:269] 1023 02:05:21.841919 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.043246 (+201327us) service_pool.cc:225] Handling call
1023 02:05:25.009723 (+2966477us) spinlock_profiling.cc:243] Waited 925 ms on lock 0x7b54000bead4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.009954 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2127532288}
W20251023 02:05:25.026999 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.026938 22000 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:25.026787 21958 rpcz_store.cc:276] 1023 02:05:18.942319 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.006995 (+6064676us) service_pool.cc:225] Handling call
1023 02:05:25.007435 (+ 440us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.019992 (+ 12557us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.020235 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.018771 22030 rpcz_store.cc:276] 1023 02:05:22.260092 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.812526 (+2552434us) service_pool.cc:225] Handling call
1023 02:05:24.812862 (+ 336us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.813177 (+ 315us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:05:24.813531 (+ 354us) log.cc:844] Serialized 14 byte log entry
1023 02:05:25.017301 (+203770us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:25.017356 (+ 55us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:25.017600 (+ 244us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.017664 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:25.017765 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":705,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"thread_start_us":179024,"threads_started":1}
I20251023 02:05:25.029461 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:25.030912 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:24.976799 24465 consensus_queue.cc:1059] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20251023 02:05:24.902210 24600 raft_consensus.cc:697] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:25.033923 22106 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:24.952417 24418 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:25.035259 22036 raft_consensus.cc:2393] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:25.005782 24597 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.036053 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:25.030597 22076 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:25.037616 21957 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
W20251023 02:05:25.040069 22077 leader_election.cc:336] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 2.466s (SENT)
W20251023 02:05:25.040601 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.039080 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:25.042838 22622 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:25.044878 24440 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:25.048931 24591 raft_consensus.cc:3055] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:25.049301 24591 raft_consensus.cc:740] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:25.050132 24591 consensus_queue.cc:260] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.050783 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:25.051498 24591 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:25.052778 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:25.053380 21957 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:25.054387 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:25.054566 24467 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.055739 21957 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:25.056262 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1568) took 4570 ms. Trace:
I20251023 02:05:25.056514 24467 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.056553 24482 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:25.058696 24405 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.059513 24576 consensus_queue.cc:237] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.065194 24323 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.056540 21957 rpcz_store.cc:276] 1023 02:05:20.486003 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.055198 (+4569195us) service_pool.cc:225] Handling call
1023 02:05:25.055527 (+ 329us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.056066 (+ 539us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.056217 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.066104 24577 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:25.066701 24577 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.067116 21957 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:25.067677 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 187) took 3752 ms. Trace:
I20251023 02:05:25.067976 21957 rpcz_store.cc:276] 1023 02:05:21.315284 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.066473 (+3751189us) service_pool.cc:225] Handling call
1023 02:05:25.066895 (+ 422us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.067447 (+ 552us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.067625 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.037815 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:25.073837 21962 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:25.069386 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:25.077800 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:24.986397 24439 consensus_queue.cc:260] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.011515 24457 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.080964 21961 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:25.083474 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:25.082238 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 63637.5us
W20251023 02:05:25.084722 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 57351.2us
W20251023 02:05:25.080219 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:25.086802 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.084103 21955 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:25.094097 24439 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.094214 24440 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:25.095395 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1584) took 2975 ms. Trace:
I20251023 02:05:25.094213 24600 consensus_queue.cc:237] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.095801 21962 rpcz_store.cc:276] 1023 02:05:22.119697 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.094455 (+2974758us) service_pool.cc:225] Handling call
1023 02:05:25.094803 (+ 348us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.095065 (+ 262us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:25.095089 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:25.095147 (+ 58us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:25.095179 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.095235 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:25.095330 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.097299 21962 raft_consensus.cc:1240] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:25.097880 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1585) took 2977 ms. Trace:
I20251023 02:05:25.098546 22110 raft_consensus.cc:2393] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:25.098428 21962 rpcz_store.cc:276] 1023 02:05:22.120018 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.096692 (+2976674us) service_pool.cc:225] Handling call
1023 02:05:25.097127 (+ 435us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.097668 (+ 541us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.097822 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:25.102182 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.075243 24577 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:25.084488 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:25.104362 24204 consensus_meta.cc:287] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.519s user 0.009s sys 0.006s
I20251023 02:05:25.134100 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:25.135823 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 45826.2us
I20251023 02:05:25.105259 24193 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.140118 21962 raft_consensus.cc:1275] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251023 02:05:25.115056 24467 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:25.141273 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 133426us
I20251023 02:05:25.141831 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1586) took 3021 ms. Trace:
I20251023 02:05:25.102367 24418 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.141973 21958 raft_consensus.cc:2393] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 7: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:25.114760 24482 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.143296 21962 rpcz_store.cc:276] 1023 02:05:22.120287 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.099813 (+2979526us) service_pool.cc:225] Handling call
1023 02:05:25.137313 (+ 37500us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.141540 (+ 4227us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.141772 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.141958 22034 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:25.075090 24597 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:25.144388 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 139934us
I20251023 02:05:25.144574 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:25.146857 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 128561us
I20251023 02:05:25.146510 22028 raft_consensus.cc:2376] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 2. Re-sending same reply.
I20251023 02:05:25.148288 24482 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.149310 22000 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:25.150355 24482 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.150940 24482 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:25.151288 24482 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:25.151679 24482 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.152005 24483 consensus_queue.cc:1048] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:05:25.152534 24583 consensus_queue.cc:1059] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:05:25.154083 21955 raft_consensus.cc:1240] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:25.154844 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1583) took 3035 ms. Trace:
I20251023 02:05:25.155248 21955 rpcz_store.cc:276] 1023 02:05:22.119437 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.102178 (+2982741us) service_pool.cc:225] Handling call
1023 02:05:25.147735 (+ 45557us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.154564 (+ 6829us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.154781 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:25.159075 21908 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7ba6e9fe4bb44e559397893fa9d899db, TS 02c53ba06cc3445a8534a84aa76b1c14 -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:25.159826 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7ba6e9fe4bb44e559397893fa9d899db, TS 02c53ba06cc3445a8534a84aa76b1c14 -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:25.161891 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.163728 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1563) took 5008 ms. Trace:
I20251023 02:05:25.163972 24611 consensus_queue.cc:1059] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251023 02:05:25.164160 21953 rpcz_store.cc:276] 1023 02:05:20.155314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.806367 (+4651053us) service_pool.cc:225] Handling call
1023 02:05:25.010128 (+203761us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.019785 (+ 9657us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.163506 (+143721us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54000bead0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.163640 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":426441984}
W20251023 02:05:25.166131 21956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1245) took 2911 ms (client timeout 1930 ms). Trace:
W20251023 02:05:25.166524 21956 rpcz_store.cc:269] 1023 02:05:22.254618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.694683 (+440065us) service_pool.cc:225] Handling call
1023 02:05:25.165761 (+2471078us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b540006ed54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.165920 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":962758400}
I20251023 02:05:25.168068 21956 raft_consensus.cc:1240] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:25.168745 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1593) took 2628 ms. Trace:
I20251023 02:05:25.170222 21959 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:25.170976 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 254) took 2246 ms. Trace:
I20251023 02:05:25.171504 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1206) took 6362 ms. Trace:
I20251023 02:05:25.171710 24597 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.172013 21959 rpcz_store.cc:276] 1023 02:05:18.809180 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.730882 (+4921702us) service_pool.cc:225] Handling call
1023 02:05:25.169789 (+1438907us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.170701 (+ 912us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.171316 (+ 615us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b540006ed50. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.171426 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3202813184}
I20251023 02:05:25.174364 22002 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:25.171334 22107 rpcz_store.cc:276] 1023 02:05:22.924246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.338778 (+1414532us) service_pool.cc:225] Handling call
1023 02:05:24.452828 (+114050us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.459034 (+ 6206us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.170758 (+711724us) spinlock_profiling.cc:243] Waited 51.4 ms on lock 0x7b54002fa3d0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.170896 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":118306048}
I20251023 02:05:25.176152 22002 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:25.177119 24294 consensus_queue.cc:1059] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:25.179721 24294 consensus_queue.cc:1243] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20251023 02:05:25.181028 24453 raft_consensus.cc:3055] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:25.181439 24453 raft_consensus.cc:740] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:25.182384 24453 consensus_queue.cc:260] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.183887 24453 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:25.177753 24204 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251023 02:05:25.184276 24604 raft_consensus.cc:3055] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:25.158830 24440 consensus_queue.cc:1059] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20251023 02:05:25.184662 24604 raft_consensus.cc:740] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:25.158885 24193 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.178275 24310 consensus_queue.cc:1059] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251023 02:05:25.179133 24467 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:25.185426 24440 raft_consensus.cc:3055] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:24.351904 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:05:25.169126 21956 rpcz_store.cc:276] 1023 02:05:22.540605 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.167417 (+2626812us) service_pool.cc:225] Handling call
1023 02:05:25.167845 (+ 428us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.168488 (+ 643us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.168689 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.173518 21959 raft_consensus.cc:1240] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:25.194593 22076 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:25.191526 21953 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:25.195667 21962 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251023 02:05:25.197239 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1243) took 3019 ms. Trace:
I20251023 02:05:25.197680 21962 rpcz_store.cc:276] 1023 02:05:22.177984 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.154892 (+2976908us) service_pool.cc:225] Handling call
1023 02:05:25.195401 (+ 40509us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.197003 (+ 1602us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.197189 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.198753 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1591) took 2884 ms. Trace:
I20251023 02:05:25.199092 21953 rpcz_store.cc:276] 1023 02:05:22.314311 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.165090 (+2850779us) service_pool.cc:225] Handling call
1023 02:05:25.165403 (+ 313us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.191372 (+ 25969us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b5400389b20. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa10d10 00007f9c5fa9ad50 00007f9c5fa9bdc1 00007f9c5fa9892f 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c
1023 02:05:25.196148 (+ 4776us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.198699 (+ 2551us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51338496}
I20251023 02:05:25.142290 24291 consensus_queue.cc:1059] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20251023 02:05:25.202437 21961 raft_consensus.cc:1275] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251023 02:05:25.202153 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1246) took 2495 ms. Trace:
I20251023 02:05:25.202936 21959 rpcz_store.cc:276] 1023 02:05:22.700942 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.173022 (+2472080us) service_pool.cc:225] Handling call
1023 02:05:25.173345 (+ 323us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.195500 (+ 22155us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.195758 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.193804 24467 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:25.196751 21858 catalog_manager.cc:5649] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 2 to 4, leader changed from 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 4 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:25.205945 24452 consensus_queue.cc:1059] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20251023 02:05:25.194033 24440 raft_consensus.cc:740] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:25.217024 24440 consensus_queue.cc:260] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.218343 21956 raft_consensus.cc:1275] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:25.218391 24440 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:25.219153 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1595) took 2333 ms. Trace:
I20251023 02:05:25.219570 22032 raft_consensus.cc:2410] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20251023 02:05:25.185550 24604 consensus_queue.cc:260] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.220546 24582 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.221056 24582 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.187351 24186 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.219570 21956 rpcz_store.cc:276] 1023 02:05:22.885554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.201675 (+2316121us) service_pool.cc:225] Handling call
1023 02:05:25.217987 (+ 16312us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.218891 (+ 904us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.219090 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.236527 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 247) took 3258 ms. Trace:
W20251023 02:05:25.237452 24547 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.995s user 0.013s sys 0.020s
I20251023 02:05:25.240245 24578 consensus_queue.cc:260] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.241274 22035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1480) took 2306 ms. Trace:
I20251023 02:05:25.213163 24467 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
W20251023 02:05:25.222803 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.222201 21954 raft_consensus.cc:2410] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251023 02:05:25.244598 22035 rpcz_store.cc:276] 1023 02:05:22.934980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.595397 (+1660417us) service_pool.cc:225] Handling call
1023 02:05:24.637971 (+ 42574us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.867849 (+229878us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.241050 (+373201us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b540001e0d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:25.241197 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":386912512}
I20251023 02:05:25.242750 22037 rpcz_store.cc:276] 1023 02:05:21.978565 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:21.978700 (+ 135us) service_pool.cc:225] Handling call
1023 02:05:25.021535 (+3042835us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.024961 (+ 3426us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.236302 (+211341us) spinlock_profiling.cc:243] Waited 3.03 s on lock 0x7b5400280d50. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.236449 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6975838464}
W20251023 02:05:25.244282 24547 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.103s user 0.014s sys 0.020s
I20251023 02:05:25.245877 21958 raft_consensus.cc:1240] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9. Ops: []
W20251023 02:05:25.248198 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.248741 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1587) took 3069 ms. Trace:
I20251023 02:05:25.249137 21958 rpcz_store.cc:276] 1023 02:05:22.178878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.146626 (+2967748us) service_pool.cc:225] Handling call
1023 02:05:25.213367 (+ 66741us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.248497 (+ 35130us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.248677 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.248994 24547 raft_consensus.cc:697] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:25.251410 21958 raft_consensus.cc:1240] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: [3.6-4.7]
I20251023 02:05:25.251682 24547 consensus_queue.cc:237] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.252988 21958 raft_consensus.cc:1275] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251023 02:05:25.253667 24602 consensus_queue.cc:1059] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251023 02:05:25.254228 24465 consensus_queue.cc:1059] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:25.255024 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:25.256038 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 49423.5us
I20251023 02:05:25.256942 22002 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:25.237496 24525 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:25.258023 22002 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:25.258569 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 195) took 2180 ms. Trace:
I20251023 02:05:25.258837 22002 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:25.258949 21959 rpcz_store.cc:276] 1023 02:05:23.078157 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.224706 (+2146549us) service_pool.cc:225] Handling call
1023 02:05:25.257974 (+ 33268us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.258256 (+ 282us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:25.258283 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:25.258321 (+ 38us) raft_consensus.cc:1645] Marking committed up to 7
1023 02:05:25.258347 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.258397 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:25.258516 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.250198 24610 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:25.260648 21956 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20251023 02:05:25.253930 21955 raft_consensus.cc:1275] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251023 02:05:25.261011 21956 pending_rounds.cc:85] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 5
I20251023 02:05:25.261269 21956 pending_rounds.cc:107] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20251023 02:05:25.261369 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1588) took 3072 ms. Trace:
I20251023 02:05:25.261459 21957 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251023 02:05:25.261495 21956 raft_consensus.cc:2889] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251023 02:05:25.262840 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 100790us
I20251023 02:05:25.263024 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1632) took 3080 ms. Trace:
I20251023 02:05:25.263860 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 196) took 1285 ms. Trace:
I20251023 02:05:25.265738 24457 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:25.265972 24295 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:25.266443 24295 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:25.266826 24295 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:25.268127 24465 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:25.268508 24465 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:25.268891 24465 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.269984 24425 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:25.270571 24425 raft_consensus.cc:697] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:25.271358 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 194) took 2193 ms. Trace:
I20251023 02:05:25.265671 21956 rpcz_store.cc:276] 1023 02:05:24.114341 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.246555 (+1132214us) service_pool.cc:225] Handling call
1023 02:05:25.260293 (+ 13738us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.263648 (+ 3355us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.263816 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.271656 24425 consensus_queue.cc:237] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.2, Last appended by leader: 2, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.262424 24578 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:25.264379 24605 log.cc:927] Time spent T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20251023 02:05:25.263864 22076 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:25.261827 21955 rpcz_store.cc:276] 1023 02:05:22.188609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.158037 (+2969428us) service_pool.cc:225] Handling call
1023 02:05:25.224849 (+ 66812us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.261121 (+ 36272us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.261319 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:25.262121 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 192) took 3190 ms. Trace:
I20251023 02:05:25.271705 21953 rpcz_store.cc:276] 1023 02:05:23.077874 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.202058 (+2124184us) service_pool.cc:225] Handling call
1023 02:05:25.270731 (+ 68673us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.271040 (+ 309us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:25.271069 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:25.271113 (+ 44us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:25.271142 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.271199 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:25.271308 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":829,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":132}
I20251023 02:05:25.185527 24552 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.263497 22109 rpcz_store.cc:276] 1023 02:05:22.183108 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.903682 (+720574us) service_pool.cc:225] Handling call
1023 02:05:22.941930 (+ 38248us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:24.585309 (+1643379us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.262701 (+677392us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b540025bcd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:25.262954 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3714951040}
W20251023 02:05:25.287670 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.158560 24483 consensus_queue.cc:1059] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
W20251023 02:05:25.283174 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 129298us
I20251023 02:05:25.284500 21957 rpcz_store.cc:276] 1023 02:05:22.071615 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.068654 (+2997039us) service_pool.cc:225] Handling call
1023 02:05:25.260919 (+192265us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b5800002a98. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9ae20e 000000000035a3c9 00007f9c649cd030 00007f9c64a95864 00007f9c64a958d1 00007f9c64a57580 00007f9c64a41ee9 00007f9c64a41992 00007f9c5dfd6d34
1023 02:05:25.261087 (+ 168us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.261901 (+ 814us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.262075 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2819712}
W20251023 02:05:25.289346 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 46297.6us
W20251023 02:05:25.292747 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 47552.2us
I20251023 02:05:25.281826 24310 consensus_queue.cc:1059] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20251023 02:05:25.294157 21955 raft_consensus.cc:1275] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251023 02:05:25.294235 24524 consensus_queue.cc:1059] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20251023 02:05:25.296273 21959 raft_consensus.cc:1240] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:25.296486 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1582) took 3177 ms. Trace:
I20251023 02:05:25.296896 21961 rpcz_store.cc:276] 1023 02:05:22.119118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.083941 (+2964823us) service_pool.cc:225] Handling call
1023 02:05:25.190338 (+106397us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.202725 (+ 12387us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.296313 (+ 93588us) spinlock_profiling.cc:243] Waited 47.9 ms on lock 0x7b540006ed50. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.296428 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110135552}
I20251023 02:05:25.299489 24069 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:25.221599 24193 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.299295 24602 raft_consensus.cc:3055] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:25.246564 24186 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:25.341228 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 197186us
I20251023 02:05:25.419176 21858 catalog_manager.cc:5649] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 2 to 5, leader changed from 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 5 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:25.419559 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:25.428184 22109 raft_consensus.cc:1275] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20251023 02:05:25.428385 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 60435.7us
I20251023 02:05:25.301937 22001 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:25.388670 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.330312 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1597) took 2404 ms. Trace:
W20251023 02:05:25.312170 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 48377.6us
W20251023 02:05:25.351881 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.432008 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:25.432515 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.283135 24610 raft_consensus.cc:697] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:25.339214 21958 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251023 02:05:25.442414 22076 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:25.443339 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 149595us
W20251023 02:05:25.433050 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.288504 24525 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:25.467242 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.466665 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:25.363075 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 54403.6us
W20251023 02:05:25.333640 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.466439 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:25.469128 22102 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:25.430302 21962 rpcz_store.cc:276] 1023 02:05:22.925854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.213995 (+2288141us) service_pool.cc:225] Handling call
1023 02:05:25.249808 (+ 35813us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.301261 (+ 51453us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:25.301291 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:25.329983 (+ 28692us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:25.330036 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.330130 (+ 94us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:25.330253 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12732,"raft.queue_time_us":40499,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":185}
W20251023 02:05:25.470608 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.470427 22076 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:05:25.470968 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.472352 22106 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:25.472698 21925 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:05:25.473078 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 90842.8us
W20251023 02:05:25.442272 21960 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1242) took 3292 ms (client timeout 1859 ms). Trace:
I20251023 02:05:25.443605 21858 catalog_manager.cc:5649] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 5 to 7, leader changed from 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 7 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:25.353097 24291 consensus_queue.cc:1048] T e32bb6c0622f44f8806f77eee0691856 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.123s
I20251023 02:05:25.476662 24292 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251023 02:05:25.474623 21960 rpcz_store.cc:269] 1023 02:05:22.161931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.584202 (+422271us) service_pool.cc:225] Handling call
1023 02:05:25.420175 (+2835973us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b5400107054. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.442175 (+ 22000us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3116213504}
I20251023 02:05:25.477128 24292 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:25.287386 24482 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:05:25.479907 24467 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.374454 24368 consensus_queue.cc:1048] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.112s
I20251023 02:05:25.481017 24613 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251023 02:05:25.386956 24609 raft_consensus.cc:3055] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:25.374687 24543 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.480222 24482 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.378578 24346 raft_consensus.cc:3055] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:25.305044 24405 raft_consensus.cc:3055] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:25.382977 24552 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.407886 24575 raft_consensus.cc:3055] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:25.383683 24604 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:25.308853 24607 log.cc:927] Time spent T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.155s user 0.004s sys 0.000s
I20251023 02:05:25.409099 24439 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.092s
I20251023 02:05:25.387266 24582 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:25.330832 24577 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:25.491983 24609 raft_consensus.cc:740] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:25.291953 24457 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.477523 24292 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.503512 24346 raft_consensus.cc:740] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:25.330229 24514 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.389737 24418 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:25.426739 24602 raft_consensus.cc:740] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:25.469981 24452 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:25.561040 24452 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:25.306769 24524 raft_consensus.cc:3055] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:25.561746 24524 raft_consensus.cc:740] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:25.507139 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:25.562310 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.563294 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:25.564579 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:25.560520 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.565946 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:25.566732 21856 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56872 (request call id 213) took 20455 ms (client timeout 15000 ms). Trace:
W20251023 02:05:25.568642 22108 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1628) took 3461 ms (client timeout 1981 ms). Trace:
I20251023 02:05:25.568261 24602 consensus_queue.cc:260] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.570326 22623 raft_consensus.cc:1240] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:25.509064 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:25.572252 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:25.573297 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:25.524730 24525 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.574142 22111 raft_consensus.cc:2410] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20251023 02:05:25.508203 22622 raft_consensus.cc:1240] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:25.556087 24577 raft_consensus.cc:697] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:25.379194 24433 raft_consensus.cc:3055] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:25.381630 24275 consensus_queue.cc:1048] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.118s
I20251023 02:05:25.517930 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:25.575269 22111 raft_consensus.cc:1240] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251023 02:05:25.519832 21961 raft_consensus.cc:1275] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:25.514791 24575 raft_consensus.cc:740] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:25.521046 24350 raft_consensus.cc:2804] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:25.521291 24467 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.521443 24290 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election won for term 5
I20251023 02:05:25.517246 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:25.337100 24483 consensus_queue.cc:1059] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.167s
I20251023 02:05:25.406029 24295 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:25.497850 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.532657 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:25.585737 21954 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:25.533979 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:25.534430 24453 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.535415 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:25.536823 21928 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:25.501509 24469 consensus_queue.cc:1059] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20251023 02:05:25.538019 22076 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:25.521931 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:25.544811 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:25.545390 24578 consensus_queue.cc:1048] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20251023 02:05:25.545432 22032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1234) took 3454 ms (client timeout 3052 ms). Trace:
W20251023 02:05:25.546092 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 73480.4us
I20251023 02:05:25.547669 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:25.547497 24465 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.545871 22031 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20251023 02:05:25.538223 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:25.554456 22106 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:25.600325 22031 pending_rounds.cc:85] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting all ops after (but not including) 7
I20251023 02:05:25.575564 24525 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:25.559857 24347 consensus_queue.cc:1059] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.003s
I20251023 02:05:25.560132 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:25.560356 22034 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:25.560220 24479 consensus_queue.cc:1059] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.005s
I20251023 02:05:25.559899 24204 consensus_queue.cc:1048] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251023 02:05:25.523365 24543 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:25.594789 24547 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.566864 22103 raft_consensus.cc:2393] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 7: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:25.575474 24457 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.511222 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:25.520541 24507 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:25.576318 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 53795us
W20251023 02:05:25.578181 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.518752 24186 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.579749 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:25.559985 21858 catalog_manager.cc:5649] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251023 02:05:25.561425 24452 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:25.582863 24350 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:25.593413 21955 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:25.576088 24433 raft_consensus.cc:740] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:25.582912 24310 consensus_queue.cc:1059] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20251023 02:05:25.592103 24290 raft_consensus.cc:697] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
W20251023 02:05:25.594336 22032 rpcz_store.cc:269] 1023 02:05:22.091281 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:23.076953 (+985672us) service_pool.cc:225] Handling call
1023 02:05:25.527539 (+2450586us) spinlock_profiling.cc:243] Waited 2.08 s on lock 0x7b540020c1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.545361 (+ 17822us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4785849728}
I20251023 02:05:25.595568 22037 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:25.669816 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:25.669340 21858 catalog_manager.cc:5649] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 4 to 5, leader changed from 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 5 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
W20251023 02:05:25.598502 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 95115.1us
I20251023 02:05:25.600580 22031 pending_rounds.cc:107] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20251023 02:05:25.589565 24294 consensus_queue.cc:1059] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20251023 02:05:25.601192 22106 raft_consensus.cc:3055] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:25.677013 22106 raft_consensus.cc:740] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:25.504717 24405 raft_consensus.cc:740] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:25.601665 24418 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:25.602743 22035 raft_consensus.cc:1240] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20251023 02:05:25.601419 21956 raft_consensus.cc:2410] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
W20251023 02:05:25.603673 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 60002.4us
I20251023 02:05:25.565712 24514 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:25.569056 22108 rpcz_store.cc:269] 1023 02:05:22.107096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:22.109055 (+ 1959us) service_pool.cc:225] Handling call
1023 02:05:25.568308 (+3459253us) spinlock_profiling.cc:243] Waited 3.45 s on lock 0x7b54000fcb54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.568446 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7932489472}
I20251023 02:05:25.576494 24577 consensus_queue.cc:237] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.623486 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
W20251023 02:05:25.650962 22000 connection.cc:232] RPC call timeout handler was delayed by -1.4870571860000155s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19810000000000003s for a response
W20251023 02:05:25.639664 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.653286 22102 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:25.612865 24602 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:25.676381 24431 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.657985 24507 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:25.674731 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 159373us
I20251023 02:05:25.668156 24350 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:25.567180 21856 rpcz_store.cc:269] 1023 02:05:05.111800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:05.125778 (+ 13978us) service_pool.cc:225] Handling call
1023 02:05:25.566485 (+20440707us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:05:15.491498 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:05:16.288083 (+796585us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:05:16.288177 (+ 94us) write_op.cc:435] Acquired schema lock
1023 02:05:16.288198 (+ 21us) tablet.cc:660] Decoding operations
1023 02:05:16.628994 (+340796us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:05:16.634970 (+ 5976us) write_op.cc:641] Partition lock acquired for write op
1023 02:05:16.674091 (+ 39121us) tablet.cc:683] Acquiring locks for 19 operations
1023 02:05:16.700712 (+ 26621us) tablet.cc:699] Row locks acquired
1023 02:05:16.700735 (+ 23us) write_op.cc:260] PREPARE: finished
1023 02:05:16.700923 (+ 188us) write_op.cc:270] Start()
1023 02:05:16.701087 (+ 164us) write_op.cc:276] Timestamp: P: 1761185116700850 usec, L: 0
1023 02:05:16.701118 (+ 31us) op_driver.cc:348] REPLICATION: starting
1023 02:05:16.701862 (+ 744us) log.cc:844] Serialized 7440 byte log entry
1023 02:05:17.107068 (+405206us) op_driver.cc:464] REPLICATION: finished
1023 02:05:18.414713 (+1307645us) write_op.cc:301] APPLY: starting
1023 02:05:18.415040 (+ 327us) tablet.cc:1365] starting BulkCheckPresence
1023 02:05:18.425347 (+ 10307us) tablet.cc:1368] finished BulkCheckPresence
1023 02:05:18.435337 (+ 9990us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:05:18.967625 (+532288us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:05:18.968037 (+ 412us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=19
1023 02:05:18.968105 (+ 68us) write_op.cc:312] APPLY: finished
1023 02:05:18.968541 (+ 436us) log.cc:844] Serialized 132 byte log entry
1023 02:05:19.107277 (+138736us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:05:19.124854 (+ 17577us) write_op.cc:454] Released schema lock
1023 02:05:19.135265 (+ 10411us) write_op.cc:341] FINISH: Updating metrics
1023 02:05:19.162251 (+ 26986us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:05:19.162299 (+ 48us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":249,"reported_tablets":38,"child_traces":[["op",{"apply.queue_time_us":1103777,"apply.run_cpu_time_us":45710,"apply.run_wall_time_us":804364,"num_ops":19,"prepare.queue_time_us":299574,"prepare.run_cpu_time_us":27482,"prepare.run_wall_time_us":1411050,"raft.queue_time_us":170825,"raft.run_cpu_time_us":10272,"raft.run_wall_time_us":367443,"replication_time_us":337065,"thread_start_us":546175,"threads_started":5,"wal-append.queue_time_us":360732,"wal-append.run_cpu_time_us":41609,"wal-append.run_wall_time_us":2099713}]]}
I20251023 02:05:25.678225 22105 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:25.620536 24543 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:25.602618 24453 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.618155 24323 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:25.621385 24295 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:25.616330 24425 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.684584 22590 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:25.632620 24193 raft_consensus.cc:3055] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:25.687984 21858 catalog_manager.cc:5649] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 3 to 4, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2), now has a pending config: VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1), VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2), VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3), VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 4 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } } } pending_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:25.672683 22031 raft_consensus.cc:2889] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:25.699062 22076 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:25.685469 22106 consensus_queue.cc:260] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.629016 24292 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.627373 24547 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.653021 24525 consensus_queue.cc:1059] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.040s
I20251023 02:05:25.620116 24467 consensus_queue.cc:1048] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.072s
I20251023 02:05:25.701524 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:25.601328 24610 consensus_queue.cc:237] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:25.703913 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 124656us
I20251023 02:05:25.704213 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:25.623071 24524 consensus_queue.cc:260] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.716892 22035 raft_consensus.cc:2393] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:25.683835 24465 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.675902 24290 consensus_queue.cc:237] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.620771 24346 consensus_queue.cc:260] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.694876 24418 raft_consensus.cc:697] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:25.704077 24431 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.730513 24418 consensus_queue.cc:237] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.731288 24431 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:25.709872 24193 raft_consensus.cc:740] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:25.732323 24193 consensus_queue.cc:260] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.711391 24405 consensus_queue.cc:260] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.713101 21928 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:25.713572 24291 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:25.714226 22034 raft_consensus.cc:1275] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251023 02:05:25.714654 21925 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:25.735858 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 252) took 2883 ms. Trace:
I20251023 02:05:25.714874 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:25.717406 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.714000 21858 catalog_manager.cc:5649] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 2 to 3, leader changed from 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:25.709182 24439 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:25.736251 22031 rpcz_store.cc:276] 1023 02:05:22.851039 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:24.589423 (+1738384us) service_pool.cc:225] Handling call
1023 02:05:25.545502 (+956079us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:25.721878 (+176376us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:25.733808 (+ 11930us) spinlock_profiling.cc:243] Waited 907 ms on lock 0x7b540020c1d0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:25.733932 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2086569216}
I20251023 02:05:25.722555 22106 raft_consensus.cc:3060] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:25.738574 22104 raft_consensus.cc:1240] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20251023 02:05:25.707317 24578 raft_consensus.cc:3055] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:25.613965 24609 consensus_queue.cc:260] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.724088 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:25.724108 24613 consensus_queue.cc:1059] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20251023 02:05:25.745960 22029 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.727475 24186 raft_consensus.cc:3055] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Stepping down as leader of term 8
I20251023 02:05:25.707299 24543 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.728274 24433 consensus_queue.cc:260] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.719362 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:25.733320 24193 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:25.731297 24507 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:25.744408 21858 catalog_manager.cc:5649] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e reported cstate change: term changed from 1 to 2, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 2 leader_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251023 02:05:25.746449 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 51412.2us
I20251023 02:05:25.748378 24514 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.748713 24613 raft_consensus.cc:3055] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:25.738003 24291 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:25.733088 24575 consensus_queue.cc:260] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.749071 24292 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:25.750070 24611 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251023 02:05:25.751214 21928 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:25.769291 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:25.742897 24578 raft_consensus.cc:740] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:25.745178 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:25.775627 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:25.775924 22035 raft_consensus.cc:2393] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:25.761626 22106 raft_consensus.cc:1275] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
W20251023 02:05:25.762916 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 60046.4us
I20251023 02:05:25.769107 22076 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:25.774483 21928 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:25.735518 24295 consensus_queue.cc:1048] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20251023 02:05:25.762184 24323 raft_consensus.cc:2749] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:25.778525 22106 pending_rounds.cc:85] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 3
I20251023 02:05:25.752403 24186 raft_consensus.cc:740] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:25.776328 22037 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:25.762573 24558 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:25.776543 22031 raft_consensus.cc:2393] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:25.748742 24591 consensus_queue.cc:1059] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20251023 02:05:25.779052 22106 pending_rounds.cc:107] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20251023 02:05:25.743381 24425 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.769788 24291 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.776994 24524 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:25.778311 24514 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:25.815927 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:25.794059 22106 raft_consensus.cc:2889] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:25.769912 24613 raft_consensus.cc:740] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:25.816622 21925 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:25.764859 24543 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.778602 24346 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.824512 24624 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:25.824962 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:25.825793 22037 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:25.824994 24624 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:25.827219 24624 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.764358 24439 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.766726 24623 consensus_queue.cc:1048] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20251023 02:05:25.793280 24433 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:25.827630 24350 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.828528 24310 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:25.836833 24310 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.838165 24439 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:25.839093 24310 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.840291 24405 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:25.814468 24368 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:25.835736 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:25.833390 22111 raft_consensus.cc:1240] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20251023 02:05:25.794082 24292 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:25.844468 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:25.829807 24575 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:25.850302 22002 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:25.845285 22031 raft_consensus.cc:2376] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7. Re-sending same reply.
I20251023 02:05:25.852640 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:25.853324 22037 raft_consensus.cc:2393] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:25.853924 22002 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:25.857793 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:25.858376 21961 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.859609 21962 raft_consensus.cc:1275] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:25.861318 21925 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:25.863399 21925 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:25.845180 24465 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:25.867635 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:25.857352 24310 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:25.844800 24469 raft_consensus.cc:2804] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:25.866593 21928 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:25.869417 21958 raft_consensus.cc:1240] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251023 02:05:25.846810 24368 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:25.868120 21960 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251023 02:05:25.849860 24576 raft_consensus.cc:3055] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:25.872836 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:25.851313 24292 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:25.873878 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:25.862493 24295 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:25.958079 24547 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:25.958431 24295 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:25.959015 24295 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:25.915778 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:25.963164 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:25.971501 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.875622 22002 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:25.914701 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:25.859221 24186 consensus_queue.cc:260] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.925693 24292 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:25.869968 24457 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:25.926177 24626 log.cc:927] Time spent T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20251023 02:05:25.925776 24613 consensus_queue.cc:260] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:25.927300 24543 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.927361 22029 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:25.933789 22616 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:25.934512 24425 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:26.040403 24630 log.cc:927] Time spent T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20251023 02:05:25.845935 24578 consensus_queue.cc:260] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.936287 22030 raft_consensus.cc:2410] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20251023 02:05:25.870604 24439 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:26.041776 24439 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:26.042141 24609 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:26.042204 24439 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:26.042738 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.043375 22030 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:26.044191 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.044798 22030 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:26.047832 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:25.862139 24558 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.080s
I20251023 02:05:25.942248 22031 raft_consensus.cc:2410] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251023 02:05:26.051119 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:25.945727 24627 consensus_queue.cc:1059] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.122s
I20251023 02:05:26.052161 24578 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 3 FOLLOWER]: Advancing to term 4
W20251023 02:05:25.947227 24625 log.cc:927] Time spent T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20251023 02:05:26.054205 24439 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.055063 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.054754 24452 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.055636 22033 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:26.056334 24439 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:26.056427 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:25.949383 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:26.057272 24452 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:25.862376 24453 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:25.950124 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.063897 22028 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:25.951292 24350 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:25.951820 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 83770.4us
I20251023 02:05:25.952706 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:26.069898 22621 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
W20251023 02:05:25.978019 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 102473us
W20251023 02:05:26.071776 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.986280 24346 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:26.074635 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:25.981995 21961 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:25.976889 21958 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:25.985062 22035 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:25.874341 24323 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251023 02:05:25.997295 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:26.002087 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.985692 24368 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:25.915016 24576 raft_consensus.cc:740] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:26.019490 21955 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:26.037036 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
W20251023 02:05:26.037577 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.039814 24624 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:25.940717 22032 raft_consensus.cc:2410] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20251023 02:05:26.121451 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:26.122185 22621 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:26.123107 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:25.876001 24514 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.189s
I20251023 02:05:26.057544 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.052158 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:26.059104 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:26.009721 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 143230us
I20251023 02:05:26.082407 22075 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:26.083288 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:26.023833 24193 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:26.130836 22031 raft_consensus.cc:2376] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6. Re-sending same reply.
W20251023 02:05:26.131275 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 261119us
W20251023 02:05:26.132380 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 50079.9us
I20251023 02:05:26.010346 24186 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:26.114722 22035 raft_consensus.cc:1275] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
W20251023 02:05:26.119225 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:25.968217 24469 raft_consensus.cc:697] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:25.862226 24482 consensus_queue.cc:1048] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.137s
I20251023 02:05:26.125785 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:26.139230 21958 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:26.123718 22028 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:25.925894 24465 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.120779 24613 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:26.136300 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.140317 21960 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251023 02:05:26.135052 22035 pending_rounds.cc:85] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting all ops after (but not including) 3
I20251023 02:05:26.144538 22035 pending_rounds.cc:107] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251023 02:05:26.143937 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.122673 24346 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:26.144881 22035 raft_consensus.cc:2889] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:26.143962 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.145296 21925 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:26.145000 22076 leader_election.cc:336] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.859s (SENT)
I20251023 02:05:26.179694 22030 raft_consensus.cc:1275] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251023 02:05:26.142149 24591 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.047s
W20251023 02:05:26.186972 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.191308 22037 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
W20251023 02:05:26.192322 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 55263.1us
I20251023 02:05:26.193499 22076 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:26.147291 22029 raft_consensus.cc:1240] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20251023 02:05:26.131134 24193 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:26.196115 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 50894.6us
W20251023 02:05:26.237924 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 44457.6us
I20251023 02:05:26.239234 22002 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:26.240222 22002 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:26.197686 24439 raft_consensus.cc:2764] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20251023 02:05:26.198091 24469 consensus_queue.cc:237] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.190644 24637 consensus_queue.cc:1048] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.009s
I20251023 02:05:26.201334 24457 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:26.202443 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 63725.2us
W20251023 02:05:26.202997 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.198688 24346 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.142213 24577 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:26.204201 24591 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251023 02:05:26.204412 22029 raft_consensus.cc:1275] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20251023 02:05:26.204909 24629 log.cc:927] Time spent T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.381s user 0.001s sys 0.000s
I20251023 02:05:26.216578 24292 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.218839 24368 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.224781 21958 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:26.225894 24576 consensus_queue.cc:260] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.227681 24624 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:26.228752 22036 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:26.230010 22621 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:26.234120 24291 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.235764 22622 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:26.237934 21925 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:26.155175 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.249712 22002 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:26.250885 24452 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:26.253870 24433 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election won for term 7
W20251023 02:05:26.296598 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 58756.3us
I20251023 02:05:26.254877 24350 raft_consensus.cc:3055] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:26.258746 24295 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.261444 24465 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:26.261991 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 113946us
I20251023 02:05:26.248318 24453 consensus_queue.cc:1048] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20251023 02:05:26.248625 24193 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:26.195464 24558 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:26.276679 22033 raft_consensus.cc:2393] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:26.284579 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.289183 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:26.294535 22002 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:26.279866 24418 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:26.300920 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:26.300932 21928 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:26.301885 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.300218 24624 raft_consensus.cc:2764] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20251023 02:05:26.268429 24577 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:26.303848 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:26.315456 22035 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:26.294133 24452 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:26.316970 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.296437 24482 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:26.323690 22030 raft_consensus.cc:1240] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:26.325313 24186 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:26.314249 24547 consensus_queue.cc:1048] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20251023 02:05:26.326954 24547 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:26.327440 24547 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.328078 24405 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:26.328548 24405 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.316672 24577 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:26.329833 24465 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:26.330273 24465 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.331609 24186 raft_consensus.cc:697] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:26.332630 24186 consensus_queue.cc:237] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.322794 22623 raft_consensus.cc:1240] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:26.335341 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:26.338007 22031 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251023 02:05:26.337394 24641 raft_consensus.cc:2955] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index 2 to 6, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:26.338913 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:26.339689 22031 raft_consensus.cc:2410] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20251023 02:05:26.340539 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:26.298115 24350 raft_consensus.cc:740] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:26.342034 22029 raft_consensus.cc:1275] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251023 02:05:26.346640 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:26.349581 21955 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:26.350737 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:26.351676 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:26.352411 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:26.341569 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:26.355151 22031 raft_consensus.cc:2376] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7. Re-sending same reply.
W20251023 02:05:26.333386 24524 consensus_meta.cc:287] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.511s user 0.005s sys 0.005s
I20251023 02:05:26.298048 24425 consensus_queue.cc:1059] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20251023 02:05:26.310170 24433 raft_consensus.cc:697] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:26.353427 24482 raft_consensus.cc:697] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:26.370491 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:26.375213 22034 raft_consensus.cc:1275] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20251023 02:05:26.323262 24452 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:26.390551 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.391559 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:26.476730 22104 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:26.392431 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:26.478729 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:26.434751 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.480396 22033 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:26.343075 24295 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:26.342924 24346 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:26.382228 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.479548 22102 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
W20251023 02:05:26.481767 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 119469us
I20251023 02:05:26.484153 22108 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:26.343386 24418 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.507829 22077 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:26.518280 22107 raft_consensus.cc:1275] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:26.318794 24558 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.371032 24627 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
W20251023 02:05:26.418769 24575 consensus_meta.cc:287] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.561s user 0.009s sys 0.001s
I20251023 02:05:26.485561 22034 pending_rounds.cc:85] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting all ops after (but not including) 5
I20251023 02:05:26.525274 22034 pending_rounds.cc:107] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20251023 02:05:26.525559 22034 raft_consensus.cc:2889] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:26.359841 24634 consensus_queue.cc:1048] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.122s
I20251023 02:05:26.447958 24323 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:26.493041 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:26.449013 24547 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:26.500128 24613 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:26.451841 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:26.506688 22105 raft_consensus.cc:1275] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20251023 02:05:26.507650 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 309415us
I20251023 02:05:26.540792 24368 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:26.541502 24368 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:26.542222 21926 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters:
I20251023 02:05:26.542066 24368 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.518148 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:26.543329 22031 raft_consensus.cc:2393] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:26.544031 24368 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:26.544122 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.544672 24368 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:26.544800 22031 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:26.545053 24368 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:26.519023 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.545444 24368 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:26.546003 22033 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:26.546838 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.547402 22033 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:26.448585 22111 raft_consensus.cc:1240] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251023 02:05:26.476614 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:26.548218 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.545749 21955 raft_consensus.cc:2468] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:26.521600 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:26.495114 22029 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:26.549858 22108 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:26.550838 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:26.551394 22108 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:26.552433 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:26.554639 22002 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:26.555711 24346 raft_consensus.cc:2804] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:26.556207 24346 raft_consensus.cc:697] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:26.557220 24346 consensus_queue.cc:237] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.562222 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.564276 22031 raft_consensus.cc:1275] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 2. (term mismatch)
I20251023 02:05:26.564746 22031 pending_rounds.cc:85] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0: Aborting all ops after (but not including) 1
I20251023 02:05:26.565084 22031 pending_rounds.cc:107] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251023 02:05:26.565413 22031 raft_consensus.cc:2889] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:26.565834 22031 pending_rounds.cc:107] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251023 02:05:26.553385 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:26.566190 24368 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.566977 22108 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:26.568029 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:26.680069 22104 raft_consensus.cc:2410] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20251023 02:05:26.549165 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:26.549239 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.566143 22031 raft_consensus.cc:2889] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:26.575285 24558 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:26.569921 24547 consensus_queue.cc:1048] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20251023 02:05:26.535326 24627 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.576100 22030 raft_consensus.cc:1275] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251023 02:05:26.720221 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:05:26.577975 22035 raft_consensus.cc:1275] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251023 02:05:26.578387 24507 raft_consensus.cc:2764] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20251023 02:05:26.579527 24642 log.cc:927] Time spent T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20251023 02:05:26.582520 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:26.728467 22105 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:26.586594 22110 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251023 02:05:26.588277 24465 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:26.588865 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:26.730044 21955 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
W20251023 02:05:26.590476 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:26.597225 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 415022us
I20251023 02:05:26.535012 24624 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:26.732040 24624 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.734088 24624 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:26.604524 22102 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
W20251023 02:05:26.613274 24638 log.cc:927] Time spent T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.270s user 0.001s sys 0.000s
I20251023 02:05:26.572949 24291 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:26.598251 24292 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:26.663269 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.568935 24323 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.571288 24613 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.613364 24602 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20251023 02:05:26.790484 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:26.791333 22104 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
W20251023 02:05:26.793642 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.682348 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:26.681353 22033 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
W20251023 02:05:26.694492 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:26.714650 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.614722 24482 consensus_queue.cc:237] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.735972 22102 raft_consensus.cc:1240] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251023 02:05:26.771342 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:26.773216 22028 raft_consensus.cc:2393] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:26.770866 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:26.802287 22031 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:26.745880 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.769863 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:26.803100 22035 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:26.798146 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.537356 24350 consensus_queue.cc:260] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.2, Last appended by leader: 2, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.721308 24275 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:26.804349 22002 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:26.805773 22111 raft_consensus.cc:1275] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 2. Preceding OpId from leader: term: 8 index: 3. (index mismatch)
I20251023 02:05:26.636327 24433 consensus_queue.cc:237] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:26.741673 24625 log.cc:927] Time spent T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
W20251023 02:05:26.819483 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.821686 22030 raft_consensus.cc:1240] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
W20251023 02:05:26.780606 24631 log.cc:927] Time spent T e32bb6c0622f44f8806f77eee0691856 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
W20251023 02:05:26.824517 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:26.737915 24610 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:26.728516 24368 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:26.792572 24439 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:26.799567 24291 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:26.840431 22033 raft_consensus.cc:1240] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: [6.2-6.2]
I20251023 02:05:26.848305 22028 raft_consensus.cc:1240] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:26.848693 22035 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
W20251023 02:05:26.842923 22032 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.007s sys 0.015s
I20251023 02:05:26.843488 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:26.850070 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:26.850783 21954 raft_consensus.cc:2393] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:26.850848 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:26.851593 21955 raft_consensus.cc:2368] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6.
I20251023 02:05:26.851804 21954 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:26.852311 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:26.852706 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:26.853307 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:26.853922 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:26.854189 21955 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:26.855002 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:26.810159 24575 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:26.856189 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:26.855031 22032 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s user 0.007s sys 0.015s
W20251023 02:05:26.838349 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 467422us
W20251023 02:05:27.339275 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 831535us
I20251023 02:05:27.340214 22077 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:26.905750 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:26.907250 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1627) took 1049 ms. Trace:
I20251023 02:05:27.342456 21961 raft_consensus.cc:2393] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:27.343945 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:27.344691 21961 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:26.860101 24291 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:26.819700 24346 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:26.905337 24655 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:05:27.250279 22587 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.009s sys 0.002s
W20251023 02:05:26.951103 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 475289us
W20251023 02:05:27.418524 22587 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.681s user 0.010s sys 0.003s
W20251023 02:05:27.418972 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 681762us
I20251023 02:05:27.083077 24323 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:27.420486 22587 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:26.831468 24637 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.348307 24575 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:26.902918 21954 raft_consensus.cc:2393] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 9: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:27.085016 24425 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:27.147526 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:27.342326 21953 rpcz_store.cc:276] 1023 02:05:25.857341 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.865152 (+ 7811us) service_pool.cc:225] Handling call
1023 02:05:25.916106 (+ 50954us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:25.916384 (+ 278us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:25.916420 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:25.919547 (+ 3127us) log.cc:844] Serialized 26 byte log entry
1023 02:05:26.241914 (+322367us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:26.241968 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:26.242060 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:26.906778 (+664718us) raft_consensus.cc:1685] finished
1023 02:05:26.906807 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:26.907169 (+ 362us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":206033,"threads_started":1,"wal-append.queue_time_us":313065}
W20251023 02:05:27.315652 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 483820us
I20251023 02:05:26.847887 24610 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:27.351120 24295 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.353212 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1626) took 1496 ms. Trace:
I20251023 02:05:26.859161 24439 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.356051 24405 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:27.356689 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:27.447892 22621 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20251023 02:05:27.448222 21955 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:27.358163 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1669) took 1014 ms. Trace:
I20251023 02:05:27.448822 22107 rpcz_store.cc:276] 1023 02:05:26.343428 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.343823 (+ 395us) service_pool.cc:225] Handling call
1023 02:05:26.372582 (+ 28759us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:26.518875 (+146293us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:27.357854 (+838979us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54002fadd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:27.358087 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":334066944}
I20251023 02:05:27.359889 24576 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:27.158123 24604 consensus_queue.cc:1059] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.203s
I20251023 02:05:27.465055 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:27.430629 24640 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.469377 21955 raft_consensus.cc:2410] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251023 02:05:27.364598 24623 raft_consensus.cc:3055] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Stepping down as leader of term 6
W20251023 02:05:27.237536 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:27.250356 24652 log.cc:927] Time spent T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20251023 02:05:27.245666 24613 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:26.926088 24649 log.cc:927] Time spent T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
W20251023 02:05:27.081692 24645 log.cc:927] Time spent T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.237s user 0.001s sys 0.000s
W20251023 02:05:27.391144 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 51011.9us
I20251023 02:05:26.838788 24418 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:26.966040 24465 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.143652 24368 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.396006 21928 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:26.965771 24647 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.436s
I20251023 02:05:27.399925 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1622) took 1543 ms. Trace:
I20251023 02:05:27.424576 24655 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.401268 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:27.412433 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:27.473790 24627 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:27.429036 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 52639.5us
I20251023 02:05:27.391801 24292 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:27.512146 21959 rpcz_store.cc:276] 1023 02:05:25.856170 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.864061 (+ 7891us) service_pool.cc:225] Handling call
1023 02:05:25.914998 (+ 50937us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:26.142608 (+227610us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:26.142662 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:26.273869 (+131207us) log.cc:844] Serialized 26 byte log entry
1023 02:05:26.667127 (+393258us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:26.667442 (+ 315us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:26.667703 (+ 261us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:27.399627 (+731924us) raft_consensus.cc:1685] finished
1023 02:05:27.399659 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:27.399850 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24278,"raft.queue_time_us":107046,"raft.run_cpu_time_us":55,"raft.run_wall_time_us":55,"thread_start_us":506290,"threads_started":2,"wal-append.queue_time_us":1029913}
W20251023 02:05:27.421402 24648 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.016s sys 0.022s
I20251023 02:05:27.436924 22622 raft_consensus.cc:1240] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:27.419184 24346 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:27.531737 24346 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:27.437867 21957 rpcz_store.cc:276] 1023 02:05:25.857085 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.863762 (+ 6677us) service_pool.cc:225] Handling call
1023 02:05:25.864214 (+ 452us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:25.864637 (+ 423us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:25.864690 (+ 53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:25.878063 (+ 13373us) log.cc:844] Serialized 52 byte log entry
1023 02:05:26.241141 (+363078us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:26.719384 (+478243us) log.cc:844] Serialized 14 byte log entry
1023 02:05:26.719971 (+ 587us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:26.720276 (+ 305us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:27.085516 (+365240us) raft_consensus.cc:1685] finished
1023 02:05:27.085545 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:27.353137 (+267592us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24796,"prepare.queue_time_us":725040,"prepare.run_cpu_time_us":8236,"prepare.run_wall_time_us":69502,"raft.queue_time_us":4690,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":20694,"thread_start_us":684627,"threads_started":2,"wal-append.queue_time_us":346769}
I20251023 02:05:27.533772 24679 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:27.534569 21957 raft_consensus.cc:1275] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251023 02:05:27.516351 24604 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.535383 24604 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.535995 21957 raft_consensus.cc:1240] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:27.451337 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:27.456152 22104 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.013s sys 0.021s
I20251023 02:05:27.537230 24604 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:27.472106 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:27.433982 24547 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.448921 24507 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.481177 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:27.491348 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:27.507256 22077 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:27.576548 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:27.473446 24543 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s user 0.004s sys 0.033s
W20251023 02:05:27.513446 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 117528us
I20251023 02:05:27.583988 21928 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:27.515633 22105 consensus_meta.cc:287] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.772s user 0.011s sys 0.000s
I20251023 02:05:27.585368 22105 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:27.586067 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1677) took 1013 ms. Trace:
I20251023 02:05:27.586460 22105 rpcz_store.cc:276] 1023 02:05:26.572621 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.582265 (+ 9644us) service_pool.cc:225] Handling call
1023 02:05:27.585995 (+1003730us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:27.450304 24295 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:27.377933 24662 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.442333 24558 consensus_queue.cc:1059] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20251023 02:05:27.597074 24662 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:27.597568 24469 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:27.598068 24469 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:27.598265 24657 log.cc:927] Time spent T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20251023 02:05:27.598439 24469 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:27.519941 24418 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:27.601958 24418 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.519484 24624 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:27.603758 24624 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:27.604107 24624 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:27.472699 24623 raft_consensus.cc:740] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:27.606402 24623 consensus_queue.cc:260] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:27.607702 24623 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:27.511552 24613 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:27.609941 24613 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.611462 24295 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:27.520370 22108 consensus_meta.cc:287] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.779s user 0.009s sys 0.000s
I20251023 02:05:27.472992 24677 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:27.436224 24425 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.525269 22623 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 8: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:27.434760 24637 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:27.525959 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:27.478060 24640 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:27.529466 24439 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:27.530463 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1630) took 1672 ms. Trace:
I20251023 02:05:27.530519 24610 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:27.668391 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:27.516094 24465 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.522305 24661 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20251023 02:05:27.534672 24646 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.534446 24679 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:27.537626 22104 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.014s sys 0.021s
I20251023 02:05:27.168995 24552 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.194s
I20251023 02:05:27.537799 21954 raft_consensus.cc:2393] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:05:27.427928 24577 consensus_meta.cc:287] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.098s user 0.010s sys 0.001s
I20251023 02:05:27.516297 24368 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:27.539337 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1542) took 1183 ms. Trace:
I20251023 02:05:27.539819 24457 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251023 02:05:27.692397 24457 raft_consensus.cc:2804] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251023 02:05:27.692577 24452 consensus_meta.cc:287] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.192s user 0.011s sys 0.000s
I20251023 02:05:27.542390 24575 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:27.542752 24648 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s user 0.016s sys 0.022s
I20251023 02:05:27.452530 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:27.703318 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1635) took 1383 ms. Trace:
I20251023 02:05:27.704012 21961 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 8: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:27.545882 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1631) took 1473 ms. Trace:
I20251023 02:05:27.704277 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:27.546016 24658 log.cc:927] Time spent T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.295s user 0.001s sys 0.000s
W20251023 02:05:27.518292 24609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s user 0.007s sys 0.007s
I20251023 02:05:27.545246 24346 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:27.554345 24350 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:27.715006 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 190538us
I20251023 02:05:27.557091 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 267) took 1595 ms. Trace:
I20251023 02:05:27.558326 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1502) took 1074 ms. Trace:
I20251023 02:05:27.537235 21957 raft_consensus.cc:1275] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251023 02:05:27.620016 22108 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
W20251023 02:05:27.658819 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:27.528945 24641 consensus_meta.cc:287] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.189s user 0.008s sys 0.002s
I20251023 02:05:27.661471 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1637) took 1316 ms. Trace:
W20251023 02:05:27.662467 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 79864us
I20251023 02:05:27.679308 24637 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:27.673919 21928 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:27.668084 21962 rpcz_store.cc:276] 1023 02:05:25.858372 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:25.935698 (+ 77326us) service_pool.cc:225] Handling call
1023 02:05:25.936090 (+ 392us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:26.128904 (+192814us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:26.128941 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:26.191136 (+ 62195us) log.cc:844] Serialized 26 byte log entry
1023 02:05:26.458135 (+266999us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:26.458190 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:26.458278 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:27.530260 (+1071982us) raft_consensus.cc:1685] finished
1023 02:05:27.530280 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:27.530406 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32792,"raft.queue_time_us":133388,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":111,"thread_start_us":352949,"threads_started":2,"wal-append.queue_time_us":436413}
W20251023 02:05:27.674662 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 190747us
W20251023 02:05:27.722766 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 48965.3us
I20251023 02:05:27.722996 22077 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:27.692873 24457 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:27.723399 21928 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:27.678257 22104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1550) took 1156 ms. Trace:
W20251023 02:05:27.723563 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 291633us
I20251023 02:05:27.689163 22076 leader_election.cc:400] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6.
I20251023 02:05:27.725304 22076 leader_election.cc:403] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20251023 02:05:27.679409 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 114127us
W20251023 02:05:27.697727 22003 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.039s sys 0.059s
I20251023 02:05:27.667968 24295 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.721637 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:27.701702 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:27.726986 21962 raft_consensus.cc:1275] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251023 02:05:27.727331 22030 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:27.727447 21954 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
W20251023 02:05:27.676290 24578 consensus_meta.cc:287] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.624s user 0.011s sys 0.001s
I20251023 02:05:27.703717 21958 rpcz_store.cc:276] 1023 02:05:26.320106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.320258 (+ 152us) service_pool.cc:225] Handling call
1023 02:05:26.320540 (+ 282us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:26.326129 (+ 5589us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:26.326159 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:26.326670 (+ 511us) log.cc:844] Serialized 26 byte log entry
1023 02:05:26.661909 (+335239us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:26.661947 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:26.662012 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:27.703108 (+1041096us) raft_consensus.cc:1685] finished
1023 02:05:27.703132 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:27.703265 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1234,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"thread_start_us":334868,"threads_started":1,"wal-append.queue_time_us":1113658}
I20251023 02:05:27.705114 21959 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:27.704579 21956 rpcz_store.cc:276] 1023 02:05:26.072956 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.073134 (+ 178us) service_pool.cc:225] Handling call
1023 02:05:26.121760 (+ 48626us) raft_consensus.cc:1471] Updating replica for 3 ops
1023 02:05:26.138169 (+ 16409us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:26.138215 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 3 ops
1023 02:05:26.180158 (+ 41943us) log.cc:844] Serialized 78 byte log entry
1023 02:05:26.551563 (+371405us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:26.928801 (+377238us) log.cc:844] Serialized 14 byte log entry
1023 02:05:26.953249 (+ 24448us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:26.953359 (+ 110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:27.545653 (+592294us) raft_consensus.cc:1685] finished
1023 02:05:27.545674 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:27.545830 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15368,"prepare.queue_time_us":759381,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":29115,"raft.queue_time_us":42146,"raft.run_cpu_time_us":470,"raft.run_wall_time_us":157420,"thread_start_us":747666,"threads_started":2,"wal-append.queue_time_us":560516}
I20251023 02:05:27.712294 21953 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:27.613188 24547 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.613040 24507 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:27.555745 24291 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:27.716662 22037 rpcz_store.cc:276] 1023 02:05:26.483915 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.814771 (+330856us) service_pool.cc:225] Handling call
1023 02:05:26.841233 (+ 26462us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:26.841480 (+ 247us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:26.841519 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:26.897558 (+ 56039us) log.cc:844] Serialized 26 byte log entry
1023 02:05:27.424629 (+527071us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:27.424666 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:27.424733 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:27.558074 (+133341us) raft_consensus.cc:1685] finished
1023 02:05:27.558101 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:27.558257 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":265781,"threads_started":1,"wal-append.queue_time_us":578735}
I20251023 02:05:27.669128 21955 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:27.720539 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1548) took 1228 ms. Trace:
I20251023 02:05:27.721560 22035 raft_consensus.cc:1275] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251023 02:05:27.748842 22108 rpcz_store.cc:276] 1023 02:05:26.491875 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.553175 (+ 61300us) service_pool.cc:225] Handling call
1023 02:05:27.717327 (+1164152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:27.699976 24439 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
W20251023 02:05:27.726086 22003 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.039s sys 0.059s
I20251023 02:05:27.689764 22109 rpcz_store.cc:276] 1023 02:05:26.356133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.373141 (+ 17008us) service_pool.cc:225] Handling call
1023 02:05:26.448791 (+ 75650us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:26.742090 (+293299us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:26.742133 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:26.797041 (+ 54908us) log.cc:844] Serialized 26 byte log entry
1023 02:05:27.296076 (+499035us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:27.296137 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:27.296231 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:27.539124 (+242893us) raft_consensus.cc:1685] finished
1023 02:05:27.539146 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:27.539284 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":96798,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"thread_start_us":726122,"threads_started":2,"wal-append.queue_time_us":706960}
I20251023 02:05:27.729470 22029 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:27.728281 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:27.732393 21928 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:27.753674 21961 raft_consensus.cc:2410] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251023 02:05:27.732889 21956 raft_consensus.cc:1275] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251023 02:05:27.703263 24452 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:27.728852 24457 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:27.733785 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:27.715631 22032 rpcz_store.cc:276] 1023 02:05:25.962181 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.130682 (+168501us) service_pool.cc:225] Handling call
1023 02:05:26.140159 (+ 9477us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:26.236437 (+ 96278us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:26.236483 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:26.266120 (+ 29637us) log.cc:844] Serialized 26 byte log entry
1023 02:05:27.339297 (+1073177us) raft_consensus.cc:1645] Marking committed up to 7
1023 02:05:27.339338 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:27.339415 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:27.554862 (+215447us) raft_consensus.cc:1685] finished
1023 02:05:27.554890 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:27.556782 (+ 1892us) spinlock_profiling.cc:243] Waited 9.1 ms on lock 0x7b540020c1d0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:27.557021 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":95893,"raft.queue_time_us":96007,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":5832,"spinlock_wait_cycles":20918528,"thread_start_us":1053781,"threads_started":1,"wal-append.queue_time_us":1273622}
I20251023 02:05:27.719323 21960 rpcz_store.cc:276] 1023 02:05:26.345437 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.346714 (+ 1277us) service_pool.cc:225] Handling call
1023 02:05:26.370545 (+ 23831us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:26.477909 (+107364us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:26.477958 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:26.501073 (+ 23115us) log.cc:844] Serialized 26 byte log entry
1023 02:05:27.346314 (+845241us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:27.346370 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:27.346455 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:27.661208 (+314753us) raft_consensus.cc:1685] finished
1023 02:05:27.661236 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:27.661401 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70187,"raft.queue_time_us":70566,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":32644,"thread_start_us":472568,"threads_started":1,"wal-append.queue_time_us":948159}
I20251023 02:05:27.740816 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:27.747085 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:27.741640 22107 raft_consensus.cc:2955] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 2 to 6, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
W20251023 02:05:27.747638 22031 consensus_meta.cc:287] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.937s user 0.007s sys 0.002s
I20251023 02:05:27.749869 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:27.523275 24668 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.742508 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:27.729664 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 62057.4us
W20251023 02:05:27.755545 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 117233us
I20251023 02:05:27.724057 22104 rpcz_store.cc:276] 1023 02:05:26.522388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.792134 (+269746us) service_pool.cc:225] Handling call
1023 02:05:26.816901 (+ 24767us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:26.817220 (+ 319us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:05:26.817245 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:26.822111 (+ 4866us) log.cc:844] Serialized 26 byte log entry
1023 02:05:27.677816 (+855705us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:27.677872 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:27.677963 (+ 91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:27.678021 (+ 58us) raft_consensus.cc:1685] finished
1023 02:05:27.678041 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:27.678200 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":87799,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":152,"thread_start_us":750288,"threads_started":1,"wal-append.queue_time_us":690602}
I20251023 02:05:27.761121 21928 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:27.702718 24577 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:27.757900 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:27.729664 24543 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:27.758204 21958 raft_consensus.cc:1275] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251023 02:05:27.762650 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:27.758301 24613 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:27.764350 21960 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:27.760512 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:27.779935 22035 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 4.
I20251023 02:05:27.764750 24634 consensus_queue.cc:1048] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:05:27.714764 24482 consensus_queue.cc:1059] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.041s
I20251023 02:05:27.765818 22623 raft_consensus.cc:3055] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:27.791029 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:27.766237 24627 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:05:27.766860 24323 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:27.767305 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:27.804894 22002 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:27.805243 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:27.805955 22035 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:27.768209 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:27.733086 24295 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.768798 24668 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:27.751868 24610 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 6
I20251023 02:05:27.729473 24575 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:27.769872 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:27.760852 22028 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:27.757089 22000 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:27.771200 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:27.733976 24637 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:27.758706 22031 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:27.782379 22037 raft_consensus.cc:2410] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20251023 02:05:27.770305 22621 raft_consensus.cc:2955] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 2 to 6, 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:27.743853 24662 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:27.785288 21956 raft_consensus.cc:1275] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 2. (index mismatch)
I20251023 02:05:27.786657 22036 raft_consensus.cc:2393] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:27.693220 24646 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.799218 22103 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 6. Current term is 7. Ops: []
W20251023 02:05:27.806675 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 83760.9us
I20251023 02:05:27.761515 24433 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:27.809332 21958 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251023 02:05:27.768976 24547 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:27.780644 24543 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:27.773479 24655 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:27.756769 24439 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.787680 24667 consensus_queue.cc:1059] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.068s
I20251023 02:05:27.811955 21955 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:28.159272 22076 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:28.097836 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:27.852841 21960 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251023 02:05:28.160105 21962 raft_consensus.cc:1240] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20251023 02:05:27.859709 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:28.171367 22035 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:27.922186 24368 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:28.161348 22076 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:27.947419 24576 consensus_queue.cc:1059] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.217s
I20251023 02:05:28.162400 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:27.921547 24695 log.cc:927] Time spent T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.165s user 0.002s sys 0.001s
I20251023 02:05:27.869344 22030 raft_consensus.cc:2393] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:27.812718 22590 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:27.925439 24689 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:28.163868 24439 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:28.164285 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:28.183813 24689 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:27.887339 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:28.165026 22031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55770 (request call id 273) took 1564 ms (client timeout 2010 ms). Trace:
I20251023 02:05:28.184260 24689 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:28.184464 22031 rpcz_store.cc:269] 1023 02:05:26.600253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.720659 (+120406us) service_pool.cc:225] Handling call
1023 02:05:28.164974 (+1444315us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:28.184916 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 372295us
W20251023 02:05:28.134970 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:27.889168 24507 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:28.185081 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:28.186723 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:28.161695 24433 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:28.185071 22028 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:28.165944 24346 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:28.165858 24575 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:27.797413 22623 raft_consensus.cc:740] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:28.115463 24452 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:28.166617 24547 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:28.167601 22109 raft_consensus.cc:2410] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20251023 02:05:27.817440 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:28.202256 22104 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:27.831816 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:27.814168 21957 raft_consensus.cc:1275] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251023 02:05:28.086773 22106 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:28.203411 22031 raft_consensus.cc:1240] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 5. Current term is 6. Ops: []
W20251023 02:05:27.858811 24687 log.cc:927] Time spent T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20251023 02:05:27.779897 24694 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:28.174721 24295 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:28.079077 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 308485us
W20251023 02:05:27.815194 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:28.172500 24469 consensus_meta.cc:287] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.573s user 0.012s sys 0.000s
I20251023 02:05:28.190852 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:28.246090 22037 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 8.
I20251023 02:05:28.133828 24482 consensus_queue.cc:1243] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer ca1362b9d9b34073bf35b6f6c9d1159a log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251023 02:05:27.858489 24686 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:28.193081 24693 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:28.193567 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 395557us
I20251023 02:05:28.264768 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:27.807121 24418 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:28.267477 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 524020us
I20251023 02:05:28.205811 24655 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:28.134389 24577 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:28.197841 22590 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:27.901095 24465 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:28.198987 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 436721us
I20251023 02:05:27.889956 24425 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:28.173044 24543 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:28.134572 24679 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:28.176988 22077 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:28.216732 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:27.900772 22617 raft_consensus.cc:2410] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20251023 02:05:28.242077 22111 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20251023 02:05:28.291287 22000 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:28.246114 22108 raft_consensus.cc:1240] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: [6.2-6.2]
I20251023 02:05:28.251370 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:28.294735 22028 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:28.249552 22587 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:27.818359 24624 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:28.297004 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:28.297523 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:28.298422 22621 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:28.262990 22104 raft_consensus.cc:1275] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251023 02:05:28.265398 22618 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:28.201880 24547 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:28.265566 24346 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20251023 02:05:28.259287 24193 consensus_meta.cc:287] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.916s user 0.008s sys 0.006s
I20251023 02:05:28.278052 21928 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:28.247514 24668 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:28.246423 24576 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251023 02:05:28.290331 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 92596us
I20251023 02:05:28.314877 22590 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:28.315644 22590 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:28.315884 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:28.293543 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:28.298786 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 49316.2us
I20251023 02:05:28.300133 22001 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:28.308140 22000 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:28.308156 21928 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:28.272650 24275 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:28.316581 21955 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:28.236253 24507 raft_consensus.cc:3055] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:28.324546 22108 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251023 02:05:28.319909 21928 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:28.330684 21928 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:28.331344 21928 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:28.342584 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:28.261345 24469 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:28.351581 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:28.353183 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:28.359495 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:28.359448 21958 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 4.
I20251023 02:05:28.367102 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:28.367887 21953 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:28.303103 24291 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:28.303790 24613 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:28.306720 24547 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:28.375743 21956 raft_consensus.cc:1240] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:28.376433 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:28.306927 24647 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:28.386293 21957 raft_consensus.cc:1275] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 2. Preceding OpId from leader: term: 8 index: 3. (index mismatch)
I20251023 02:05:28.317409 24577 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:28.319203 24368 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:28.275017 24655 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:28.329038 24507 raft_consensus.cc:740] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:28.320489 24576 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:28.343209 24634 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:28.268260 24686 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:28.454933 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:28.455760 21953 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:28.343896 24552 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:28.477342 21955 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:28.395780 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1304) took 2056 ms. Trace:
W20251023 02:05:28.415016 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 114985us
I20251023 02:05:28.480106 22029 rpcz_store.cc:276] 1023 02:05:26.338889 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.550066 (+211177us) service_pool.cc:225] Handling call
1023 02:05:26.550396 (+ 330us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:27.752919 (+1202523us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:28.395564 (+642645us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54002404d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:28.395722 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2711267712}
I20251023 02:05:28.343618 24677 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:28.480829 22001 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:28.431339 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:28.435921 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 104608us
I20251023 02:05:28.310984 24465 consensus_queue.cc:1048] T d80086b174ba4cee80e75f95401994e3 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
W20251023 02:05:28.473970 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 78521.5us
I20251023 02:05:28.272338 24693 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:28.296790 22037 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:28.489334 22002 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:28.489744 22037 raft_consensus.cc:606] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:28.486542 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:28.319394 24193 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:28.490314 22037 raft_consensus.cc:614] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:28.327391 24290 consensus_queue.cc:1048] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.102s
I20251023 02:05:28.485910 21958 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:28.396795 24668 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:28.419855 24346 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:28.342617 24275 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:28.419620 24514 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251023 02:05:28.517971 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 341067us
W20251023 02:05:28.512442 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 204373us
I20251023 02:05:28.290110 22623 consensus_queue.cc:260] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:28.437863 24576 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:28.437161 24634 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:28.491427 24674 log.cc:927] Time spent T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
W20251023 02:05:28.577479 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 88220.7us
I20251023 02:05:28.457942 24662 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:28.459162 24368 raft_consensus.cc:697] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:28.477406 24552 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:28.427534 24439 consensus_queue.cc:1048] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.138s
I20251023 02:05:28.428856 24577 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:28.488402 24677 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:28.458899 24670 consensus_queue.cc:1059] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.145s
W20251023 02:05:28.709472 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 228806us
W20251023 02:05:28.943427 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:29.027387 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 349901us
I20251023 02:05:28.842288 22622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 273) took 1111 ms. Trace:
I20251023 02:05:28.522253 22615 raft_consensus.cc:1275] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20251023 02:05:29.254755 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:28.888031 24707 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:28.819945 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:29.256017 24624 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.255847 24707 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.116703 24514 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
W20251023 02:05:29.067420 22109 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.007s sys 0.005s
I20251023 02:05:28.828857 24740 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:29.257885 22109 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.915s user 0.011s sys 0.006s
W20251023 02:05:29.117518 22036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1295) took 2814 ms (client timeout 2224 ms). Trace:
W20251023 02:05:29.257684 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 1.55675e+06us
I20251023 02:05:28.928105 21953 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:29.259419 21925 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:29.106833 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1273) took 1820 ms. Trace:
W20251023 02:05:28.711050 22588 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:28.888056 22624 consensus_meta.cc:287] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.701s user 0.009s sys 0.001s
W20251023 02:05:28.643177 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:29.216398 22102 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.006s sys 0.010s
I20251023 02:05:29.261968 22624 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
W20251023 02:05:29.262530 22102 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.012s user 0.007s sys 0.010s
I20251023 02:05:28.845893 22033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1512) took 1124 ms. Trace:
I20251023 02:05:29.260985 21925 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:05:29.260759 22036 rpcz_store.cc:269] 1023 02:05:26.302892 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.303150 (+ 258us) service_pool.cc:225] Handling call
1023 02:05:29.117227 (+2814077us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b540026f554. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:29.117415 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3377766656}
I20251023 02:05:29.263022 22033 rpcz_store.cc:276] 1023 02:05:27.721539 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.721679 (+ 140us) service_pool.cc:225] Handling call
1023 02:05:27.747777 (+ 26098us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:27.758986 (+ 11209us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:27.759015 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:27.759460 (+ 445us) log.cc:844] Serialized 26 byte log entry
1023 02:05:28.179080 (+419620us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:28.179117 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:28.179177 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:28.845673 (+666496us) raft_consensus.cc:1685] finished
1023 02:05:28.845706 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:28.845834 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31344,"raft.queue_time_us":11188,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"thread_start_us":374193,"threads_started":1,"wal-append.queue_time_us":1065298}
W20251023 02:05:29.216230 22107 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.271s user 0.004s sys 0.012s
W20251023 02:05:29.268167 22107 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.323s user 0.005s sys 0.013s
I20251023 02:05:29.269254 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1553) took 1548 ms. Trace:
W20251023 02:05:28.840127 24673 log.cc:927] Time spent T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 1.243s user 0.002s sys 0.004s
I20251023 02:05:29.269641 22107 rpcz_store.cc:276] 1023 02:05:27.720464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.720594 (+ 130us) service_pool.cc:225] Handling call
1023 02:05:27.741301 (+ 20707us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:27.741508 (+ 207us) raft_consensus.cc:1534] Early marking committed up to index 6
1023 02:05:27.944833 (+203325us) log.cc:844] Serialized 14 byte log entry
1023 02:05:29.268903 (+1324070us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:29.268961 (+ 58us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:29.269008 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:29.269082 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:29.269198 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":122,"thread_start_us":1323400,"threads_started":1}
I20251023 02:05:29.254716 24662 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:28.933018 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 452271us
I20251023 02:05:29.271476 24662 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:29.271581 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1495) took 2920 ms. Trace:
W20251023 02:05:29.272797 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:29.258237 24439 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:29.255044 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 671147us
I20251023 02:05:29.284829 22000 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:29.260723 22002 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:29.031180 22104 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.049s sys 0.103s
W20251023 02:05:29.286049 22104 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.841s user 0.051s sys 0.115s
W20251023 02:05:29.027208 22074 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.011s sys 0.001s
W20251023 02:05:29.286947 22074 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.017s user 0.015s sys 0.003s
W20251023 02:05:28.673287 24292 consensus_meta.cc:287] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.107s user 0.009s sys 0.002s
I20251023 02:05:28.823372 24405 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:29.258155 24740 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:28.666136 24425 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.600s
I20251023 02:05:29.266651 24634 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:29.256664 24624 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:29.278609 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 695269us
I20251023 02:05:28.660663 24575 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:28.642819 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:28.566913 24668 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:29.280005 22624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 277) took 1099 ms. Trace:
I20251023 02:05:29.260330 21962 rpcz_store.cc:276] 1023 02:05:27.286104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.727710 (+441606us) service_pool.cc:225] Handling call
1023 02:05:28.159871 (+432161us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:28.170210 (+ 10339us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:29.106544 (+936334us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b54000c3850. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:29.106746 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":993217920}
I20251023 02:05:29.272230 22034 rpcz_store.cc:276] 1023 02:05:26.351409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.575356 (+223947us) service_pool.cc:225] Handling call
1023 02:05:26.575736 (+ 380us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:26.576064 (+ 328us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:26.576090 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:26.576566 (+ 476us) log.cc:844] Serialized 26 byte log entry
1023 02:05:26.861193 (+284627us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:27.369791 (+508598us) log.cc:844] Serialized 14 byte log entry
1023 02:05:27.370406 (+ 615us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:27.370477 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:28.839454 (+1468977us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:29.271214 (+431760us) raft_consensus.cc:1685] finished
1023 02:05:29.271345 (+ 131us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:29.271512 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":65703,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":196400,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":204,"spinlock_wait_cycles":37771136,"thread_start_us":749452,"threads_started":2,"wal-append.queue_time_us":1020408}
W20251023 02:05:29.281646 22587 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.955s user 0.002s sys 0.017s
I20251023 02:05:28.838883 24323 raft_consensus.cc:3055] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Stepping down as leader of term 6
W20251023 02:05:28.925750 22111 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.009s sys 0.000s
W20251023 02:05:29.288731 21956 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.006s sys 0.029s
I20251023 02:05:29.291085 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:28.712705 24704 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.296s user 0.001s sys 0.001s
I20251023 02:05:28.703083 24291 raft_consensus.cc:697] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:29.256798 24514 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
W20251023 02:05:28.640882 24350 consensus_meta.cc:287] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.922s user 0.009s sys 0.000s
I20251023 02:05:29.109661 24578 raft_consensus.cc:3055] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Stepping down as leader of term 8
I20251023 02:05:29.254055 22622 rpcz_store.cc:276] 1023 02:05:27.730407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.730543 (+ 136us) service_pool.cc:225] Handling call
1023 02:05:27.754251 (+ 23708us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:27.754613 (+ 362us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:27.754645 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:27.756585 (+ 1940us) log.cc:844] Serialized 26 byte log entry
1023 02:05:27.900502 (+143917us) raft_consensus.cc:1645] Marking committed up to 7
1023 02:05:27.900561 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:27.900656 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:28.842080 (+941424us) raft_consensus.cc:1685] finished
1023 02:05:28.842099 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:28.842225 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2442,"raft.queue_time_us":2398,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":12205,"thread_start_us":135849,"threads_started":1,"wal-append.queue_time_us":1076317}
I20251023 02:05:28.666879 24507 consensus_queue.cc:260] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:28.580242 24646 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:28.837582 24731 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:28.823196 24701 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:29.075212 24469 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:28.618237 24694 raft_consensus.cc:697] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:29.122202 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:29.118573 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:29.095631 24640 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.113386 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:28.674753 24627 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:29.275063 24552 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:29.497638 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 262) took 3209 ms. Trace:
I20251023 02:05:29.498116 22103 rpcz_store.cc:276] 1023 02:05:26.288051 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.288199 (+ 148us) service_pool.cc:225] Handling call
1023 02:05:26.296482 (+ 8283us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:28.175207 (+1878725us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:29.497330 (+1322123us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54001cf2d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:29.497555 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3455589632}
W20251023 02:05:29.375916 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:29.500844 22035 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.173s user 0.004s sys 0.001s
W20251023 02:05:29.376216 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 115579us
I20251023 02:05:28.837888 24728 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:29.252545 24193 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:28.836539 24725 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:28.641851 24433 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:28.711948 24688 raft_consensus.cc:2749] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:29.110831 24647 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:29.286737 24457 consensus_meta.cc:287] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.507s user 0.011s sys 0.000s
W20251023 02:05:28.711021 21912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 44445.3us
I20251023 02:05:28.845785 24693 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:29.410512 22589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.534s (SENT) [suppressed 27 similar messages]
I20251023 02:05:28.599855 24679 consensus_queue.cc:1059] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.788s
I20251023 02:05:29.558732 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:29.559552 22028 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:28.581431 24346 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.274850 24677 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:29.287588 24275 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.293009 24418 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:29.293356 24439 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:29.413790 22587 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.087s user 0.002s sys 0.018s
W20251023 02:05:29.567739 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 1.24142e+06us
W20251023 02:05:29.434659 21956 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s user 0.006s sys 0.030s
I20251023 02:05:29.570221 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:29.418968 22111 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.032s user 0.014s sys 0.003s
I20251023 02:05:29.323894 24292 raft_consensus.cc:2749] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:29.413077 22624 rpcz_store.cc:276] 1023 02:05:28.163698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:28.186492 (+ 22794us) service_pool.cc:225] Handling call
1023 02:05:29.262700 (+1076208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:29.325245 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:29.443531 22077 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:29.294582 24609 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:29.294340 24669 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:05:29.451135 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 190197us
I20251023 02:05:29.410796 24722 raft_consensus.cc:2749] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:29.470330 22106 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56894 (request call id 264) took 3181 ms (client timeout 1871 ms). Trace:
W20251023 02:05:29.430210 24715 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
I20251023 02:05:29.418757 24323 raft_consensus.cc:740] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:29.449659 24578 raft_consensus.cc:740] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:29.443164 24558 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.844s user 0.025s sys 0.025s
I20251023 02:05:29.613756 24669 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:29.614480 24558 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.017s user 0.025s sys 0.025s
I20251023 02:05:29.474960 24665 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:29.615232 24665 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:29.615723 24623 consensus_meta.cc:287] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 2.008s user 0.009s sys 0.006s
I20251023 02:05:29.616062 24669 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:29.424958 24707 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.617229 24665 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:29.471661 24637 consensus_meta.cc:287] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.300s user 0.010s sys 0.002s
I20251023 02:05:29.410924 24727 raft_consensus.cc:2804] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:29.450338 24667 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:29.493723 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 208983us
W20251023 02:05:29.501233 22035 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.174s user 0.004s sys 0.001s
I20251023 02:05:29.475791 24350 raft_consensus.cc:2804] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election won for term 5
W20251023 02:05:29.483600 24576 consensus_meta.cc:287] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.824s user 0.010s sys 0.000s
I20251023 02:05:29.430367 24663 consensus_queue.cc:1048] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20251023 02:05:29.419764 24624 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.622133 24577 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:29.504088 22002 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:29.522531 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 245738us
I20251023 02:05:29.547077 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:29.686874 22033 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:28.828775 24295 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:29.484505 24686 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.552943 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:29.692466 22037 raft_consensus.cc:2393] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:29.491148 24701 raft_consensus.cc:697] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:29.496929 24507 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:29.556362 24703 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:29.544072 24405 consensus_queue.cc:1048] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20251023 02:05:29.557431 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:29.523409 24543 consensus_meta.cc:287] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.204s user 0.010s sys 0.002s
I20251023 02:05:29.523439 24728 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:29.503275 24482 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:29.579855 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:29.719246 22036 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:29.545740 24647 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:29.557950 24736 log.cc:927] Time spent T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
W20251023 02:05:29.604292 22106 rpcz_store.cc:269] 1023 02:05:26.288733 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.288882 (+ 149us) service_pool.cc:225] Handling call
1023 02:05:29.470120 (+3181238us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b54001cded4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:29.470257 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3714283776}
I20251023 02:05:29.605046 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:29.605406 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1652) took 1289 ms. Trace:
I20251023 02:05:29.725399 21956 rpcz_store.cc:276] 1023 02:05:28.315622 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:28.403454 (+ 87832us) service_pool.cc:225] Handling call
1023 02:05:28.481925 (+ 78471us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:28.482180 (+ 255us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:28.666463 (+184283us) log.cc:844] Serialized 14 byte log entry
1023 02:05:29.605070 (+938607us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:29.605124 (+ 54us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:29.605172 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:29.605234 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:29.605343 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35248,"prepare.queue_time_us":351008,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":35661,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":5400,"thread_start_us":1015553,"threads_started":2,"wal-append.queue_time_us":917260}
I20251023 02:05:29.726485 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:29.609817 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 166397us
I20251023 02:05:29.611497 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:29.561831 22616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34806 (request call id 378) took 1820 ms (client timeout 2309 ms). Trace:
I20251023 02:05:29.611740 24323 consensus_queue.cc:260] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:29.484526 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 200) took 1941 ms. Trace:
W20251023 02:05:29.618281 24742 log.cc:927] Time spent T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20251023 02:05:29.765524 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:29.608281 24291 consensus_queue.cc:237] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:29.618927 22030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1513) took 1897 ms (client timeout 1605 ms). Trace:
I20251023 02:05:29.614303 24578 consensus_queue.cc:260] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.621076 22107 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:29.491009 24731 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.537648 24575 consensus_queue.cc:1059] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.051s
I20251023 02:05:29.584880 24679 consensus_queue.cc:1243] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Peer 818907d63f1f4d84ba267ebe6ee837e0 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20251023 02:05:29.614287 24721 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20251023 02:05:29.673489 22000 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:29.550135 24457 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.585487 22619 raft_consensus.cc:2410] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20251023 02:05:29.494434 24368 consensus_queue.cc:237] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.585914 24275 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:28.712515 24465 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:29.629931 24292 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:29.689919 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:29.816360 22105 raft_consensus.cc:1240] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:29.629397 24727 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:29.632844 24637 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251023 02:05:29.693030 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 282540us
I20251023 02:05:29.737807 24482 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.602404 24634 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.547011 24725 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.609309 24609 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:29.678706 24735 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:29.714268 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:29.717759 24543 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
W20251023 02:05:29.700577 24689 consensus_meta.cc:287] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.516s user 0.011s sys 0.002s
I20251023 02:05:29.687605 24350 raft_consensus.cc:697] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:29.609427 21925 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:29.727532 22103 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:29.727552 22077 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:29.857931 24613 consensus_meta.cc:287] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 1.461s user 0.009s sys 0.015s
I20251023 02:05:29.727550 22037 raft_consensus.cc:2393] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:29.758248 21961 rpcz_store.cc:276] 1023 02:05:27.543588 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.759211 (+215623us) service_pool.cc:225] Handling call
1023 02:05:27.788539 (+ 29328us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:27.788867 (+ 328us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:27.788903 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:27.799919 (+ 11016us) log.cc:844] Serialized 26 byte log entry
1023 02:05:28.164649 (+364730us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:28.164688 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:28.164760 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:29.484287 (+1319527us) raft_consensus.cc:1685] finished
1023 02:05:29.484307 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:29.484463 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":364331,"threads_started":1,"wal-append.queue_time_us":467980}
I20251023 02:05:29.704852 24686 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election won for term 7
W20251023 02:05:29.618470 24610 consensus_meta.cc:287] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.446s user 0.012s sys 0.000s
I20251023 02:05:29.880729 24610 raft_consensus.cc:2749] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6.
W20251023 02:05:29.774478 22030 rpcz_store.cc:269] 1023 02:05:27.721852 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.733535 (+ 11683us) service_pool.cc:225] Handling call
1023 02:05:29.618701 (+1885166us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54003aa554. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:29.618857 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":252584320}
I20251023 02:05:29.881829 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:29.882856 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:29.883316 22620 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:29.883544 24624 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.883919 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:29.686170 24576 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.884738 24669 raft_consensus.cc:2749] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251023 02:05:29.884409 24350 consensus_queue.cc:237] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:29.718971 24728 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
W20251023 02:05:29.824928 22619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34806 (request call id 379) took 2075 ms (client timeout 1841 ms). Trace:
I20251023 02:05:29.723047 24740 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.794401 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:29.890478 22036 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:29.725226 24668 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:29.798383 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 125220us
I20251023 02:05:29.889788 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1276) took 2473 ms. Trace:
W20251023 02:05:29.893045 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:29.694888 24295 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.871505 24439 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.681252 24667 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:29.794823 24715 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20251023 02:05:29.816881 21928 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:29.819468 22035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 277) took 2045 ms. Trace:
I20251023 02:05:29.914841 22035 rpcz_store.cc:276] 1023 02:05:27.773776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:28.176517 (+402741us) service_pool.cc:225] Handling call
1023 02:05:28.196758 (+ 20241us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:28.268633 (+ 71875us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:28.268660 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:28.269125 (+ 465us) log.cc:844] Serialized 26 byte log entry
1023 02:05:29.650110 (+1380985us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:29.650149 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:29.650245 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:29.819167 (+168922us) raft_consensus.cc:1685] finished
1023 02:05:29.819198 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:29.819384 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19646,"raft.queue_time_us":65434,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":6939,"thread_start_us":1394183,"threads_started":2,"wal-append.queue_time_us":1421689}
I20251023 02:05:29.823472 22075 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:29.690908 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 186902us
I20251023 02:05:29.682677 24694 consensus_queue.cc:237] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:29.738005 22616 rpcz_store.cc:269] 1023 02:05:27.734620 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.751480 (+ 16860us) service_pool.cc:225] Handling call
1023 02:05:29.122012 (+1370532us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b5800214598. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:05:29.555002 (+432990us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3863424}
I20251023 02:05:29.807847 24752 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:29.858014 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:29.860344 22033 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:29.932994 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:29.858275 22104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 270) took 2095 ms. Trace:
I20251023 02:05:29.818382 24292 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.815474 24275 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.941290 24732 consensus_queue.cc:1059] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20251023 02:05:29.942137 24705 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:29.942662 24705 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:29.944669 24705 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:29.879523 24731 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.793872 24346 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.813174 24465 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:29.822259 24727 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:29.945971 24465 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:29.946588 24295 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.873315 22587 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:29.949688 24609 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:29.949947 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 76724.7us
I20251023 02:05:29.950242 24609 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251023 02:05:29.874315 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 264955us
I20251023 02:05:29.950605 24609 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:29.877737 24744 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:29.951042 24609 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Advancing to term 8
W20251023 02:05:29.878110 24751 log.cc:927] Time spent T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20251023 02:05:29.878325 24760 raft_consensus.cc:2749] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:29.857800 24323 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:29.955039 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1551) took 3417 ms. Trace:
I20251023 02:05:29.879482 24686 raft_consensus.cc:697] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:29.956135 24752 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:29.880231 24696 log.cc:927] Time spent T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 2.124s user 0.000s sys 0.017s
I20251023 02:05:29.957168 24686 consensus_queue.cc:237] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.888072 21954 raft_consensus.cc:1240] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:29.958933 24752 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.959731 24543 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:29.886206 24576 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:29.884477 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 157013us
I20251023 02:05:29.961134 24186 raft_consensus.cc:3055] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:29.961534 24186 raft_consensus.cc:740] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:29.961971 22077 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:29.962435 24186 consensus_queue.cc:260] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.963601 24186 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:29.824143 24755 raft_consensus.cc:2764] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20251023 02:05:29.966064 24723 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:29.891592 22110 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:29.893011 21959 rpcz_store.cc:276] 1023 02:05:27.402446 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.733390 (+330944us) service_pool.cc:225] Handling call
1023 02:05:27.733721 (+ 331us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:27.981173 (+247452us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:27.981220 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:28.178668 (+197448us) log.cc:844] Serialized 26 byte log entry
1023 02:05:28.394728 (+216060us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:28.394768 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:28.394849 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:29.876086 (+1481237us) raft_consensus.cc:1685] finished
1023 02:05:29.876111 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:29.876254 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12887,"raft.queue_time_us":45900,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":92,"thread_start_us":344103,"threads_started":2,"wal-append.queue_time_us":1216784}
I20251023 02:05:29.966421 22028 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "774c010489aa4775bb6d636926900529"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
W20251023 02:05:29.886554 22619 rpcz_store.cc:269] 1023 02:05:27.747165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.750996 (+ 3831us) service_pool.cc:225] Handling call
1023 02:05:29.113175 (+1362179us) spinlock_profiling.cc:243] Waited 4.86 ms on lock 0x7b5800214598. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:05:29.822745 (+709570us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11167104}
I20251023 02:05:29.967799 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:29.933858 22104 rpcz_store.cc:276] 1023 02:05:27.762362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:28.307093 (+544731us) service_pool.cc:225] Handling call
1023 02:05:28.342865 (+ 35772us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:28.366156 (+ 23291us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:28.366198 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:28.366746 (+ 548us) log.cc:844] Serialized 26 byte log entry
1023 02:05:29.286717 (+919971us) raft_consensus.cc:1645] Marking committed up to 7
1023 02:05:29.487368 (+200651us) log.cc:844] Serialized 14 byte log entry
1023 02:05:29.487623 (+ 255us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:29.487735 (+ 112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:29.858043 (+370308us) raft_consensus.cc:1685] finished
1023 02:05:29.858066 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:29.858218 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":155123,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":453543,"raft.run_cpu_time_us":411,"raft.run_wall_time_us":80384,"thread_start_us":926433,"threads_started":2,"wal-append.queue_time_us":1021704}
I20251023 02:05:29.805982 22590 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:29.969030 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 163183us
I20251023 02:05:29.955438 22102 rpcz_store.cc:276] 1023 02:05:26.537878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:26.815277 (+277399us) service_pool.cc:225] Handling call
1023 02:05:26.815590 (+ 313us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:26.815899 (+ 309us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:26.815938 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:26.829840 (+ 13902us) log.cc:844] Serialized 26 byte log entry
1023 02:05:28.250854 (+1421014us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b18002f5f00. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9cdd37 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c5f62f2bb 00007f9c5f634dc3 00007f9c5f635171 00007f9c5fa5465f
1023 02:05:29.282163 (+1031309us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:29.282224 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:29.282320 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:29.954753 (+672433us) raft_consensus.cc:1685] finished
1023 02:05:29.954776 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:29.954976 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":40007,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":47228,"spinlock_wait_cycles":4469632,"thread_start_us":1011962,"threads_started":1,"wal-append.queue_time_us":2825938}
I20251023 02:05:29.968422 22103 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 3.
W20251023 02:05:29.883893 22620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34806 (request call id 377) took 2153 ms (client timeout 1605 ms). Trace:
I20251023 02:05:29.902211 24575 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:29.906600 24457 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:29.878350 24634 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.910452 22000 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:29.887604 24578 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
W20251023 02:05:29.918025 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 94635.3us
W20251023 02:05:29.923019 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 106206us
W20251023 02:05:29.928516 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:29.887248 24613 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:30.019317 22589 leader_election.cc:336] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499): Timed out: RequestConsensusVote RPC to 127.20.81.1:41499 timed out after 1.871s (SENT)
I20251023 02:05:29.874758 24689 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.922104 22002 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:29.937637 24668 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:29.938930 24647 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:29.939641 24725 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.940832 24482 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.941212 24292 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:29.942999 24275 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:29.691741 24193 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:29.857054 24701 consensus_queue.cc:237] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:29.966517 24723 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:29.895634 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:29.883136 24662 consensus_meta.cc:287] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.611s user 0.009s sys 0.000s
I20251023 02:05:29.976536 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:30.056434 22035 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 9.
W20251023 02:05:29.992302 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:30.057685 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:30.058146 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:29.792678 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 267) took 2031 ms. Trace:
I20251023 02:05:30.010233 22102 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20251023 02:05:29.972126 24736 log.cc:927] Time spent T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20251023 02:05:29.984809 24591 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:30.067848 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.069669 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.022861 24743 raft_consensus.cc:3055] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:29.968276 24439 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251023 02:05:30.027585 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 105577us
W20251023 02:05:30.024017 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:30.045341 24655 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:29.950464 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:30.029963 24760 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:30.084954 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:30.087260 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:30.088181 22108 raft_consensus.cc:2393] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:29.990778 22033 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:30.089186 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:30.089864 22108 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 8
I20251023 02:05:30.041209 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:30.030050 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 119671us
I20251023 02:05:30.094079 22000 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:30.094280 22029 raft_consensus.cc:1275] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20251023 02:05:29.935909 24547 consensus_meta.cc:287] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.491s user 0.010s sys 0.002s
I20251023 02:05:30.037519 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 274) took 2264 ms. Trace:
W20251023 02:05:30.040143 22110 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1554) took 2319 ms (client timeout 3078 ms). Trace:
I20251023 02:05:30.026295 22077 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:30.013176 22036 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:30.029769 21928 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:30.031170 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.042723 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 266) took 2281 ms. Trace:
I20251023 02:05:30.058703 22111 rpcz_store.cc:276] 1023 02:05:27.761553 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:28.307249 (+545696us) service_pool.cc:225] Handling call
1023 02:05:28.307599 (+ 350us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:28.307858 (+ 259us) raft_consensus.cc:1534] Early marking committed up to index 6
1023 02:05:29.571309 (+1263451us) log.cc:844] Serialized 14 byte log entry
1023 02:05:29.791239 (+219930us) log.cc:844] Serialized 14 byte log entry
1023 02:05:29.791899 (+ 660us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:29.792205 (+ 306us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:29.792360 (+ 155us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:29.792449 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:29.792597 (+ 148us) inbound_call.cc:173] Queueing success response
1023 02:05:30.035277 (+242680us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b1800472540. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9eb3d2 00007f9c5c9ed7a1 00007f9c5c9ed759 00007f9c5c9ed6e9 00007f9c5c9ed6b1 00007f9c5c9ec9ad 0000000000379254 0000000000379099 00007f9c5c9ce75a
Metrics: {"mutex_wait_us":35438,"prepare.queue_time_us":1126093,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":92,"raft.queue_time_us":509,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":9359,"spinlock_wait_cycles":13400192,"thread_start_us":1402794,"threads_started":2,"wal-append.queue_time_us":481294}
I20251023 02:05:30.045398 24705 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:30.111964 22103 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 8.
I20251023 02:05:30.031383 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:30.024519 22589 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:30.028107 24576 consensus_queue.cc:1048] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.048s
I20251023 02:05:30.011183 24543 consensus_queue.cc:1048] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20251023 02:05:30.102095 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:29.985934 22620 rpcz_store.cc:269] 1023 02:05:27.730743 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.732967 (+ 2224us) service_pool.cc:225] Handling call
1023 02:05:29.118358 (+1385391us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5800214598. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c9ade91 00007f9c5c9add42 00007f9c65849ad9 00007f9c649ccd30 00007f9c64a9584b 00007f9c64a958d1 00007f9c64a43660 00007f9c64a42f44 00007f9c5dfdb834 00007f9c5dfdb7c2
1023 02:05:29.883820 (+765462us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22995712}
I20251023 02:05:30.029067 24469 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:30.106137 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:30.108166 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:30.074370 24743 raft_consensus.cc:740] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:30.036064 22615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 383) took 2270 ms. Trace:
I20251023 02:05:30.031526 24748 consensus_queue.cc:1048] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
W20251023 02:05:30.110846 22110 rpcz_store.cc:269] 1023 02:05:27.720758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.720851 (+ 93us) service_pool.cc:225] Handling call
1023 02:05:30.039978 (+2319127us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b54002d0fd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:30.040087 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4884696320}
I20251023 02:05:30.113570 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.111865 22109 rpcz_store.cc:276] 1023 02:05:27.761265 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:28.251361 (+490096us) service_pool.cc:225] Handling call
1023 02:05:28.273175 (+ 21814us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:28.299295 (+ 26120us) raft_consensus.cc:1534] Early marking committed up to index 7
1023 02:05:29.258337 (+959042us) log.cc:844] Serialized 14 byte log entry
1023 02:05:29.688273 (+429936us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:29.699157 (+ 10884us) log.cc:844] Serialized 26 byte log entry
1023 02:05:29.699262 (+ 105us) raft_consensus.cc:1645] Marking committed up to 7
1023 02:05:29.699284 (+ 22us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:29.699328 (+ 44us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:30.042466 (+343138us) raft_consensus.cc:1685] finished
1023 02:05:30.042495 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:30.042656 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1120496,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":7549,"raft.queue_time_us":193887,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":37164,"thread_start_us":1344632,"threads_started":2,"wal-append.queue_time_us":661266}
I20251023 02:05:30.066809 24663 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:30.056383 24723 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:30.165205 22108 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:30.118834 24732 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:30.115408 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.116133 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.115319 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.074151 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:30.176961 21962 raft_consensus.cc:2393] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:30.064877 24193 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:30.066143 24662 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:30.120045 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.031848 24665 consensus_queue.cc:1059] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20251023 02:05:30.127036 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:30.127177 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 97489.2us
I20251023 02:05:30.127671 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.127957 21958 raft_consensus.cc:1240] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20251023 02:05:30.129217 21960 raft_consensus.cc:1275] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20251023 02:05:30.051788 24610 consensus_queue.cc:1048] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20251023 02:05:30.180776 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.134121 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:30.028204 24575 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:30.136821 21955 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:30.038498 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1274) took 2666 ms. Trace:
I20251023 02:05:30.199568 21954 rpcz_store.cc:276] 1023 02:05:27.371770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.733971 (+362201us) service_pool.cc:225] Handling call
1023 02:05:27.734213 (+ 242us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:29.958665 (+2224452us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:30.038241 (+ 79576us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b5400106b54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:30.038444 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4953197440}
I20251023 02:05:30.201010 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.139250 22000 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:30.202368 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 63206.5us
I20251023 02:05:30.122984 24547 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:30.075798 24439 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:30.140981 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.052060 24668 raft_consensus.cc:697] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:30.144135 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.110738 22032 rpcz_store.cc:276] 1023 02:05:27.772721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.772945 (+ 224us) service_pool.cc:225] Handling call
1023 02:05:28.167652 (+394707us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:28.167868 (+ 216us) raft_consensus.cc:1534] Early marking committed up to index 6
1023 02:05:28.180624 (+ 12756us) log.cc:844] Serialized 14 byte log entry
1023 02:05:29.569863 (+1389239us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b180048aa20. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9606ea 00007f9c65fd4a29 00007f9c5c820990 00007f9c5c9e84d4 00007f9c5c9e8445 00007f9c63488b27 00007f9c5faa3537 00007f9c5fab2c4f 00007f9c5fab2bc8
1023 02:05:29.702991 (+133128us) log.cc:844] Serialized 14 byte log entry
1023 02:05:29.708275 (+ 5284us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:29.708345 (+ 70us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:29.708411 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:29.708498 (+ 87us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:30.037343 (+328845us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b54002a7bd0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:30.037461 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":191319,"prepare.queue_time_us":1307093,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":71,"raft.queue_time_us":37297,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":27314,"spinlock_wait_cycles":882942592,"thread_start_us":227443,"threads_started":2,"wal-append.queue_time_us":1035322}
I20251023 02:05:30.028873 24755 consensus_queue.cc:1048] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
W20251023 02:05:30.153424 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 174412us
I20251023 02:05:30.157943 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.026664 24724 consensus_queue.cc:1048] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.108s
I20251023 02:05:30.084339 22034 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:30.066977 24591 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:30.178859 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.229177 22108 raft_consensus.cc:2393] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 9: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:30.135565 24705 raft_consensus.cc:697] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:30.238904 24705 consensus_queue.cc:237] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:30.190517 22106 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:30.192030 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.242014 24741 consensus_queue.cc:1048] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.187s
I20251023 02:05:30.193861 21928 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:29.985116 24731 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.088s
I20251023 02:05:30.195256 21957 raft_consensus.cc:2393] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:30.197365 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.203280 22000 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:30.203677 24744 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:30.159188 24482 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:30.204514 21954 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:30.186015 22616 raft_consensus.cc:2393] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 7: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:30.165282 24611 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:30.209617 22104 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:30.214953 22103 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:30.183456 24655 raft_consensus.cc:2749] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251023 02:05:30.229449 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.284132 21960 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:30.232192 22028 raft_consensus.cc:606] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:30.285116 22028 raft_consensus.cc:614] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:30.160545 24295 consensus_queue.cc:1048] T 00301d3b4a804bf19bad66c51fe19faa P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20251023 02:05:30.237046 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:30.237699 24591 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:30.239594 24765 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20251023 02:05:30.141791 24740 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:30.242461 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.243014 24714 raft_consensus.cc:2749] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:30.243681 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:30.193895 21956 raft_consensus.cc:2410] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251023 02:05:30.152877 24469 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:30.352558 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 346617us
W20251023 02:05:30.254067 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 60295.1us
I20251023 02:05:30.355880 21928 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:30.257668 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 54459.1us
I20251023 02:05:30.357223 22000 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:30.359155 22000 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:30.261232 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:30.228974 24772 log.cc:927] Time spent T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
I20251023 02:05:30.218868 24439 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:30.145237 24760 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:30.266346 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.266883 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.267549 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:30.230389 24766 log.cc:927] Time spent T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.342s user 0.001s sys 0.001s
I20251023 02:05:30.172299 24663 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:30.288152 24482 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:30.314040 24775 log.cc:927] Time spent T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20251023 02:05:30.289234 24662 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:30.286638 22076 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:30.156419 22615 rpcz_store.cc:276] 1023 02:05:27.765189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.765476 (+ 287us) service_pool.cc:225] Handling call
1023 02:05:27.765929 (+ 453us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:29.255232 (+1489303us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:30.035818 (+780586us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b5400194e54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:30.036006 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":980677504}
I20251023 02:05:30.263842 24543 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251023 02:05:30.332185 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 85024.1us
I20251023 02:05:30.405144 22077 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:30.268564 24346 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251023 02:05:30.345430 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.348670 22110 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:30.332862 24748 consensus_queue.cc:1048] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.062s
I20251023 02:05:30.350725 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:30.321535 24552 consensus_meta.cc:287] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.774s user 0.006s sys 0.002s
W20251023 02:05:30.352041 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 332780us
I20251023 02:05:30.032022 24418 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:30.362171 22587 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:30.363417 22000 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:30.455816 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:30.353271 24727 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:30.456445 22034 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:30.390023 24725 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:30.386806 22106 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:30.386018 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.387830 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.388758 21961 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:30.325860 24689 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:30.397210 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 110691us
I20251023 02:05:30.397886 21953 raft_consensus.cc:2410] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251023 02:05:30.398602 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:30.460187 21958 raft_consensus.cc:2393] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:30.399262 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.331073 24665 consensus_queue.cc:1059] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20251023 02:05:30.403636 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:30.394222 24686 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:30.405429 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.459018 24418 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:30.468245 24725 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:30.470460 24725 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:30.470808 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.471907 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:30.473069 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.473677 21956 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:30.474532 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:30.475072 21956 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:30.459053 24552 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:30.477599 24725 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:30.477967 24552 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:30.478505 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:30.479530 24725 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:30.395378 24774 log.cc:927] Time spent T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.164s user 0.001s sys 0.000s
I20251023 02:05:30.452037 22077 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:30.410110 21928 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:30.480754 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 70747.7us
I20251023 02:05:30.481631 21928 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:30.482192 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.483655 24762 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251023 02:05:30.484360 24762 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:30.484489 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.484840 24762 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:30.485291 21957 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:30.486450 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.487489 21957 raft_consensus.cc:2410] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20251023 02:05:30.349429 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.488369 24350 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:30.389806 24731 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 02:05:30.458600 21954 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:30.463254 21925 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:30.489378 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.489737 24731 consensus_queue.cc:1048] T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:30.489992 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.490697 21925 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:30.387619 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:30.491217 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.491806 21955 raft_consensus.cc:2376] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6. Re-sending same reply.
I20251023 02:05:30.492324 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.469447 24686 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:30.492637 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.492995 24686 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:30.493443 21955 raft_consensus.cc:2410] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20251023 02:05:30.407083 24346 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:30.407126 24655 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:30.496448 21925 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:30.285687 24743 consensus_queue.cc:260] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:30.497771 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 134429us
I20251023 02:05:30.516593 22000 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:30.395442 24776 log.cc:927] Time spent T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20251023 02:05:30.489992 21960 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 8.
I20251023 02:05:30.483603 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.502147 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:30.530128 22002 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:30.508512 22034 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:30.330229 24668 consensus_queue.cc:237] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:30.516592 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 154528us
W20251023 02:05:30.493021 24778 log.cc:927] Time spent T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20251023 02:05:30.519872 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:30.521929 22030 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 9.
I20251023 02:05:30.522805 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1552) took 2803 ms. Trace:
I20251023 02:05:30.524032 24743 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:30.530993 22002 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:30.531787 22590 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:30.404793 24543 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:30.454548 24576 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:30.534739 22031 raft_consensus.cc:2393] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:30.519301 24641 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20251023 02:05:30.516769 24775 log.cc:927] Time spent T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20251023 02:05:30.402556 24647 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:30.514813 24346 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Advancing to term 10
W20251023 02:05:30.676294 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 144576us
I20251023 02:05:30.539000 21956 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:30.543414 22105 rpcz_store.cc:276] 1023 02:05:27.719532 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.719672 (+ 140us) service_pool.cc:225] Handling call
1023 02:05:27.719962 (+ 290us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:29.816757 (+2096795us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:30.522543 (+705786us) spinlock_profiling.cc:243] Waited 2.1 s on lock 0x7b54000fb754. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:30.522749 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4821082368}
I20251023 02:05:30.659070 24714 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:30.536780 24725 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:30.515698 24655 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:30.592382 24575 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:30.603196 24323 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:30.511049 24744 raft_consensus.cc:2749] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:30.565310 21957 raft_consensus.cc:1275] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20251023 02:05:30.677448 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 146527us
W20251023 02:05:30.739805 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 204237us
W20251023 02:05:30.685106 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:30.666529 24543 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:30.539474 24689 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:30.533533 24552 raft_consensus.cc:2804] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251023 02:05:30.619077 24677 consensus_meta.cc:287] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.998s user 0.012s sys 0.000s
I20251023 02:05:30.681710 24714 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:30.588832 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:30.741978 22002 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:30.755383 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.758483 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.759582 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:30.740248 24507 consensus_meta.cc:287] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.026s user 0.011s sys 0.000s
I20251023 02:05:30.732620 24655 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:30.684275 24725 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:30.588243 24418 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:30.811345 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.836957 22105 raft_consensus.cc:2468] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:30.742890 24647 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:30.746358 24323 raft_consensus.cc:697] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:30.746209 24689 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:30.750078 24552 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:30.818339 22002 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:30.761364 24714 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:30.755623 24193 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:30.825266 21957 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251023 02:05:30.590267 24783 consensus_queue.cc:1059] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.154s
I20251023 02:05:30.543598 24578 consensus_queue.cc:1048] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.199s
I20251023 02:05:30.837056 24725 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:30.810829 24744 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251023 02:05:30.828749 22111 raft_consensus.cc:2410] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20251023 02:05:30.809880 24469 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:30.834864 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.757655 24732 consensus_queue.cc:1048] T 0dc4e5ba4d0d4809a19f394786d00288 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20251023 02:05:30.754981 24677 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:30.842958 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.823832 24663 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:30.826901 24575 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:30.832229 24691 raft_consensus.cc:3055] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Stepping down as leader of term 6
W20251023 02:05:30.826248 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 297997us
I20251023 02:05:30.962843 21925 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:30.885407 24727 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:30.841152 24641 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:30.841008 24667 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:30.915953 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.993432 22618 raft_consensus.cc:1240] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: []
W20251023 02:05:30.928767 24723 consensus_meta.cc:287] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.727s user 0.007s sys 0.004s
I20251023 02:05:30.993543 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.995435 22105 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
W20251023 02:05:30.916419 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 433260us
I20251023 02:05:30.846113 24760 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:30.844841 24689 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:30.848604 24552 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:30.920339 21957 raft_consensus.cc:1240] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:30.883466 24705 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:30.827790 24350 raft_consensus.cc:697] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:30.917276 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:30.917126 24744 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:30.930194 24691 raft_consensus.cc:740] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:30.998410 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.001353 21928 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:30.996256 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.008159 22589 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:31.010843 22105 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:31.013252 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.013988 22106 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:31.014896 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.015524 22106 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:31.016419 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.017215 22106 raft_consensus.cc:2393] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 7: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:30.996212 24641 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:31.024832 22034 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20251023 02:05:31.024840 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.033100 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:30.995810 24482 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:31.040349 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.042156 22106 raft_consensus.cc:1275] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251023 02:05:31.043634 22106 raft_consensus.cc:1275] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20251023 02:05:31.045495 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 231822us
I20251023 02:05:30.931494 24686 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:30.968152 24575 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:31.047994 22000 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:31.003918 24727 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:31.057898 21927 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s user 0.003s sys 0.008s
I20251023 02:05:31.009863 24744 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Advancing to term 10
W20251023 02:05:31.039024 24465 consensus_meta.cc:287] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.092s user 0.012s sys 0.000s
W20251023 02:05:31.064011 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:31.041637 24641 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:31.000576 24723 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.034246 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:31.007052 24705 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.059583 24647 raft_consensus.cc:697] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:31.122460 24647 consensus_queue.cc:237] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.125907 22102 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:31.128551 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1657) took 1012 ms. Trace:
I20251023 02:05:31.129396 21955 raft_consensus.cc:1240] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: [6.2-6.2]
I20251023 02:05:31.129016 21960 rpcz_store.cc:276] 1023 02:05:30.115789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.528879 (+413090us) service_pool.cc:225] Handling call
1023 02:05:30.529189 (+ 310us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:30.529373 (+ 184us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:30.529395 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:30.529905 (+ 510us) log.cc:844] Serialized 26 byte log entry
1023 02:05:31.007508 (+477603us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:31.007541 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.007609 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:31.128308 (+120699us) raft_consensus.cc:1685] finished
1023 02:05:31.128329 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:31.128487 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":415282,"threads_started":1,"wal-append.queue_time_us":591081}
I20251023 02:05:31.130905 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1288) took 1055 ms. Trace:
I20251023 02:05:31.131266 21953 rpcz_store.cc:276] 1023 02:05:30.090499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.115085 (+ 24586us) service_pool.cc:225] Handling call
1023 02:05:31.130716 (+1015631us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b5400301954. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.130847 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":507932928}
I20251023 02:05:31.131808 22000 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:31.134621 22000 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:31.098875 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 90805.1us
I20251023 02:05:31.070763 24482 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251023 02:05:31.137243 22589 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:31.110852 24723 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:31.111277 24591 consensus_meta.cc:287] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.746s user 0.006s sys 0.003s
I20251023 02:05:31.112389 24677 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:31.113822 24609 consensus_meta.cc:287] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 1.162s user 0.008s sys 0.003s
I20251023 02:05:31.114182 24705 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:31.115491 24728 consensus_meta.cc:287] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.230s user 0.005s sys 0.006s
W20251023 02:05:31.117250 24514 consensus_meta.cc:287] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.675s user 0.010s sys 0.003s
I20251023 02:05:31.086941 24691 consensus_queue.cc:260] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:31.118831 24186 consensus_meta.cc:287] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.155s user 0.011s sys 0.001s
I20251023 02:05:31.118938 24741 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.097942 24465 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:31.131556 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 671845us
I20251023 02:05:31.175702 22076 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:31.004014 24323 consensus_queue.cc:237] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:31.144887 24782 log.cc:927] Time spent T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.267s user 0.000s sys 0.001s
W20251023 02:05:31.138240 24785 log.cc:927] Time spent T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.320s user 0.000s sys 0.001s
I20251023 02:05:31.159240 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:31.138000 24692 raft_consensus.cc:3055] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:31.168071 22000 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:31.139555 24418 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:31.160418 24788 log.cc:927] Time spent T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.397s user 0.001s sys 0.000s
I20251023 02:05:31.161969 24748 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.188261 22000 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:31.188681 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1574) took 1072 ms. Trace:
I20251023 02:05:31.159340 24808 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:31.164089 24755 consensus_queue.cc:1059] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20251023 02:05:31.153952 24798 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:31.191462 22037 raft_consensus.cc:2368] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 8. Current term is 9.
I20251023 02:05:31.167567 24577 raft_consensus.cc:3055] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:31.155880 24807 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:31.194577 22000 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:31.190186 24691 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.160274 24609 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.179443 24741 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.166628 24728 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:31.197220 24786 log.cc:927] Time spent T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.337s user 0.000s sys 0.001s
I20251023 02:05:31.197729 24507 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:31.276178 22033 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 9: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:31.276306 24803 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.102s
I20251023 02:05:31.210213 24292 raft_consensus.cc:3055] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:31.277087 24292 raft_consensus.cc:740] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:31.177364 24457 raft_consensus.cc:2764] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20251023 02:05:31.277891 24292 consensus_queue.cc:260] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.279203 24292 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.153296 24591 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.177785 24677 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:31.280732 24677 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:31.281112 24677 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:31.208753 22076 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
W20251023 02:05:31.282310 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 73640.2us
I20251023 02:05:31.199699 24732 raft_consensus.cc:2749] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:31.283015 22076 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:31.283291 21956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1665) took 1024 ms. Trace:
I20251023 02:05:31.187422 24692 raft_consensus.cc:740] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:31.169718 24514 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:31.245021 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 50556.5us
W20251023 02:05:31.245628 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 94797.9us
I20251023 02:05:31.168943 24725 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.189985 24186 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251023 02:05:31.264493 24788 log.cc:927] Time spent T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20251023 02:05:31.265166 24575 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.211766 24577 raft_consensus.cc:740] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:31.199131 24783 consensus_queue.cc:1059] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20251023 02:05:31.270543 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1577) took 1126 ms. Trace:
I20251023 02:05:31.244261 24807 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:31.201521 24748 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.201499 24578 consensus_queue.cc:1048] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.089s
W20251023 02:05:31.198271 24799 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
W20251023 02:05:31.294806 24543 consensus_meta.cc:287] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.541s user 0.007s sys 0.003s
I20251023 02:05:31.200551 24808 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:31.211588 24743 consensus_meta.cc:287] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.531s user 0.006s sys 0.006s
W20251023 02:05:31.245831 24802 log.cc:927] Time spent T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20251023 02:05:31.293385 22076 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:31.290625 24735 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
W20251023 02:05:31.317340 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:31.321741 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:31.304021 24609 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.168309 24350 consensus_queue.cc:237] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.203024 24727 consensus_queue.cc:1048] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20251023 02:05:31.301313 22107 rpcz_store.cc:276] 1023 02:05:30.144617 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.191370 (+ 46753us) service_pool.cc:225] Handling call
1023 02:05:30.191700 (+ 330us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:30.192010 (+ 310us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:30.192039 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:30.192480 (+ 441us) log.cc:844] Serialized 26 byte log entry
1023 02:05:30.397160 (+204680us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:30.397200 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:30.397297 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:31.270263 (+872966us) raft_consensus.cc:1685] finished
1023 02:05:31.270293 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:31.270467 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":344,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":76,"thread_start_us":204309,"threads_started":1,"wal-append.queue_time_us":545954}
I20251023 02:05:31.331915 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.244725 24705 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:31.329344 24755 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:31.334640 24705 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.336745 24705 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.263417 24482 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:31.272517 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48448 (request call id 18) took 1658 ms. Trace:
I20251023 02:05:31.337762 24783 consensus_queue.cc:1243] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20251023 02:05:31.338013 22028 rpcz_store.cc:276] 1023 02:05:29.613881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:29.966210 (+352329us) service_pool.cc:225] Handling call
1023 02:05:31.272302 (+1306092us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b540020b2d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa933f3 00007f9c64a45bca 00007f9c5dfe9934 00007f9c5dfe98c2 00007f9c5dfe97f1 00007f9c5dfe976c 00007f9c5dfe8a22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.272426 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":609019136}
W20251023 02:05:31.339138 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:31.276142 24728 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:31.304857 21957 raft_consensus.cc:1240] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:31.283702 21956 rpcz_store.cc:276] 1023 02:05:30.258944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.936780 (+677836us) service_pool.cc:225] Handling call
1023 02:05:30.937161 (+ 381us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:30.937521 (+ 360us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:30.937556 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:30.938158 (+ 602us) log.cc:844] Serialized 26 byte log entry
1023 02:05:31.108575 (+170417us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:31.108652 (+ 77us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.108751 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:31.283098 (+174347us) raft_consensus.cc:1685] finished
1023 02:05:31.283118 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:31.283240 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":95092,"threads_started":1,"wal-append.queue_time_us":169559}
I20251023 02:05:31.291127 24457 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 02:05:31.296406 24507 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:31.340462 24655 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.341094 24507 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:31.341485 24507 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.341679 22108 raft_consensus.cc:2393] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 9: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:31.342228 24728 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:31.342726 24728 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.343186 24577 consensus_queue.cc:260] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.344384 24577 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.344600 24728 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:31.299611 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 889870us
I20251023 02:05:31.318578 24808 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:31.345868 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:31.347049 22619 raft_consensus.cc:2393] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 9: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:31.348019 22077 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:31.348790 24576 consensus_queue.cc:1048] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:05:31.349730 24457 consensus_queue.cc:1048] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:31.350128 24576 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:31.350695 24576 raft_consensus.cc:697] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
W20251023 02:05:31.351881 24762 consensus_meta.cc:287] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.866s user 0.009s sys 0.000s
I20251023 02:05:31.333099 22107 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:31.351540 24576 consensus_queue.cc:237] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.352849 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.304435 24807 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.353514 22107 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:31.354389 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.354931 22107 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:31.355679 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.356647 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.356797 24714 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:31.358036 24346 consensus_meta.cc:287] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.672s user 0.010s sys 0.000s
I20251023 02:05:31.357823 24762 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.358431 24346 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.358757 24725 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.359642 24762 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 10 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:31.360626 24740 consensus_meta.cc:287] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.008s user 0.011s sys 0.000s
I20251023 02:05:31.361006 24740 raft_consensus.cc:2749] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:31.361605 21962 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:31.318024 24686 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:31.363860 24439 consensus_meta.cc:287] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.969s user 0.006s sys 0.002s
I20251023 02:05:31.353212 24552 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.365859 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.321017 24735 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:31.243841 24798 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.378460 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.380179 22077 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:31.381913 22077 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:31.389775 24810 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:31.395879 24798 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:31.396996 24465 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:31.327975 24741 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.398320 22035 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.398725 22029 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 9. Current term is 10. Ops: []
I20251023 02:05:31.318136 24543 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.405395 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.406245 22107 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 9
I20251023 02:05:31.408142 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:31.408519 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:31.409104 22624 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:31.402249 24823 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:31.416440 24514 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.417860 24591 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.420665 22077 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:31.421083 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:31.421958 22028 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:31.426431 22624 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:31.427989 22001 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:31.443527 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.448580 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.459379 22077 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:31.402527 24748 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:31.461972 21959 raft_consensus.cc:2368] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7.
I20251023 02:05:31.485435 21959 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1293) took 1393 ms. Trace:
I20251023 02:05:31.485922 21959 rpcz_store.cc:276] 1023 02:05:30.092100 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.385330 (+293230us) service_pool.cc:225] Handling call
1023 02:05:31.485201 (+1099871us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54002de1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.485363 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2282091904}
I20251023 02:05:31.463755 22077 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:31.490765 22623 raft_consensus.cc:3060] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:31.493299 24748 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:31.493871 24748 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:31.493708 24507 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.494356 24748 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:31.502470 24810 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:31.502933 24810 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:31.508001 24807 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.509692 22034 raft_consensus.cc:1240] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:31.510124 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1576) took 1373 ms. Trace:
I20251023 02:05:31.510619 22110 rpcz_store.cc:276] 1023 02:05:30.136754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.157707 (+ 20953us) service_pool.cc:225] Handling call
1023 02:05:31.509916 (+1352209us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54000118d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.510050 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":289162496}
I20251023 02:05:31.512017 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:31.512425 21858 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33304 (request call id 93) took 17914 ms (client timeout 15000 ms). Trace:
I20251023 02:05:31.514055 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
W20251023 02:05:31.514488 24795 log.cc:927] Time spent T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.392s user 0.000s sys 0.003s
I20251023 02:05:31.514386 24792 consensus_queue.cc:1059] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251023 02:05:31.515183 22624 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.515401 24792 raft_consensus.cc:3055] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:31.515829 24792 raft_consensus.cc:740] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
W20251023 02:05:31.516672 22111 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1572) took 1408 ms (client timeout 1504 ms). Trace:
I20251023 02:05:31.516647 24792 consensus_queue.cc:260] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:31.517207 22111 rpcz_store.cc:269] 1023 02:05:30.108724 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.113337 (+ 4613us) service_pool.cc:225] Handling call
1023 02:05:31.516461 (+1403124us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b54000fbc54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.516614 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1542221696}
I20251023 02:05:31.512818 22110 raft_consensus.cc:2410] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20251023 02:05:31.518327 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.518579 24346 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:31.518900 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.519452 22110 raft_consensus.cc:2468] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:31.519786 21954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 204) took 1417 ms. Trace:
I20251023 02:05:31.520215 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.520874 22105 raft_consensus.cc:1275] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20251023 02:05:31.458447 24457 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:31.521389 24641 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.522287 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:31.522222 24692 consensus_queue.cc:260] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.523660 24641 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.520135 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1306) took 1245 ms. Trace:
I20251023 02:05:31.464354 24822 consensus_queue.cc:1048] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
W20251023 02:05:31.524335 21962 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36488 (request call id 205) took 1421 ms (client timeout 1534 ms). Trace:
I20251023 02:05:31.524932 24714 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.521271 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.471937 24547 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:31.475172 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:31.476318 21961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1284) took 1407 ms (client timeout 1571 ms). Trace:
I20251023 02:05:31.524338 21957 rpcz_store.cc:276] 1023 02:05:30.274445 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.010316 (+735871us) service_pool.cc:225] Handling call
1023 02:05:31.039731 (+ 29415us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:31.340463 (+300732us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.519943 (+179480us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b5400024754. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:31.520082 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":609322240}
W20251023 02:05:31.526413 21961 rpcz_store.cc:269] 1023 02:05:30.068966 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.069101 (+ 135us) service_pool.cc:225] Handling call
1023 02:05:31.476151 (+1407050us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b5400218c54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.476260 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":567925120}
I20251023 02:05:31.477030 24686 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.527235 22029 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:31.527454 24686 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.527871 22029 raft_consensus.cc:606] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:31.528591 22030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1536) took 1712 ms (client timeout 2026 ms). Trace:
I20251023 02:05:31.436570 24465 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election won for term 9
W20251023 02:05:31.528970 22030 rpcz_store.cc:269] 1023 02:05:29.816717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:29.883649 (+ 66932us) service_pool.cc:225] Handling call
1023 02:05:31.528449 (+1644800us) spinlock_profiling.cc:243] Waited 637 ms on lock 0x7b540001d1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.528537 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1465910656}
I20251023 02:05:31.529484 24686 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.529681 22000 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:31.530784 22000 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:31.532032 22000 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:31.532418 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.533061 24547 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.533485 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.534211 22110 raft_consensus.cc:2393] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 9: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:31.534451 24798 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251023 02:05:31.535075 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.535571 22110 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Advancing to term 10
W20251023 02:05:31.536370 24689 consensus_meta.cc:287] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.527s user 0.006s sys 0.007s
I20251023 02:05:31.536742 24689 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.480437 24543 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:31.482225 24797 log.cc:927] Time spent T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20251023 02:05:31.537726 24457 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:31.538329 24457 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:31.483201 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1584) took 1075 ms. Trace:
I20251023 02:05:31.539034 22102 rpcz_store.cc:276] 1023 02:05:30.408126 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.409833 (+ 1707us) service_pool.cc:225] Handling call
1023 02:05:30.457278 (+ 47445us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:31.126387 (+669109us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.482976 (+356589us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b54001ce3d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:31.483141 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1537299712}
I20251023 02:05:31.540475 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.541114 22102 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:31.518425 22615 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:31.524730 21962 rpcz_store.cc:269] 1023 02:05:30.103111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.180542 (+ 77431us) service_pool.cc:225] Handling call
1023 02:05:31.524179 (+1343637us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54000c4c54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.524279 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2683485440}
I20251023 02:05:31.541795 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.520125 21954 rpcz_store.cc:276] 1023 02:05:30.102824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.493212 (+390388us) service_pool.cc:225] Handling call
1023 02:05:30.494149 (+ 937us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:30.494361 (+ 212us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:30.494398 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:30.494864 (+ 466us) log.cc:844] Serialized 26 byte log entry
1023 02:05:30.742691 (+247827us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:30.827547 (+ 84856us) log.cc:844] Serialized 14 byte log entry
1023 02:05:31.034637 (+207090us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.034766 (+ 129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:31.514080 (+479314us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b24000ed188. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c83ffe4 00007f9c5c83e0eb 00007f9c5c8cce49 00007f9c5c8ccad5 00007f9c5c8cc11d 00007f9c5f61920e 00007f9c5f63093c 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1
1023 02:05:31.519486 (+ 5406us) raft_consensus.cc:1685] finished
1023 02:05:31.519595 (+ 109us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:31.519730 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":89729,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"spinlock_wait_cycles":39619456,"thread_start_us":454101,"threads_started":2,"wal-append.queue_time_us":626975}
I20251023 02:05:31.470592 24668 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:31.521229 22105 pending_rounds.cc:85] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 4
I20251023 02:05:31.543499 22105 pending_rounds.cc:107] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20251023 02:05:31.543807 22105 raft_consensus.cc:2889] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:31.432149 24823 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:31.546458 22110 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 10.
I20251023 02:05:31.546644 24823 raft_consensus.cc:697] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:31.547308 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.547837 22110 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:31.548314 22102 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:31.547904 24823 consensus_queue.cc:237] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.533511 24547 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.550647 24558 raft_consensus.cc:3055] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:31.551083 24547 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:31.551141 24558 raft_consensus.cc:740] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:31.552660 24558 consensus_queue.cc:260] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:31.553851 24558 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
W20251023 02:05:31.534153 22031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1518) took 2012 ms (client timeout 1812 ms). Trace:
W20251023 02:05:31.555083 22031 rpcz_store.cc:269] 1023 02:05:29.522214 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:29.523927 (+ 1713us) service_pool.cc:225] Handling call
1023 02:05:31.533910 (+2009983us) spinlock_profiling.cc:243] Waited 828 ms on lock 0x7b54002a94d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.534067 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1904644992}
I20251023 02:05:31.562135 24792 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.563047 24465 raft_consensus.cc:697] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:31.564179 22033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55770 (request call id 291) took 1086 ms. Trace:
I20251023 02:05:31.564661 22033 rpcz_store.cc:276] 1023 02:05:30.478117 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.478262 (+ 145us) service_pool.cc:225] Handling call
1023 02:05:31.563949 (+1085687us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b54002e9d54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.564111 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1832732928}
I20251023 02:05:31.567046 24552 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.377933 24439 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.570266 24692 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:31.570447 22030 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:31.572326 22030 raft_consensus.cc:606] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:31.572748 22030 raft_consensus.cc:614] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:31.556844 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.571332 22000 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:31.575973 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1582) took 1225 ms. Trace:
I20251023 02:05:31.462796 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.191905 22109 rpcz_store.cc:276] 1023 02:05:30.116569 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.158026 (+ 41457us) service_pool.cc:225] Handling call
1023 02:05:30.179146 (+ 21120us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:30.179314 (+ 168us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:30.675525 (+496211us) log.cc:844] Serialized 14 byte log entry
1023 02:05:31.188221 (+512696us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:31.188288 (+ 67us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:31.188343 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.188429 (+ 86us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:31.188614 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":290636,"prepare.queue_time_us":71392,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":9994,"raft.queue_time_us":258609,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":2334,"thread_start_us":683981,"threads_started":2,"wal-append.queue_time_us":478696}
I20251023 02:05:31.576325 22108 rpcz_store.cc:276] 1023 02:05:30.350222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.350391 (+ 169us) service_pool.cc:225] Handling call
1023 02:05:31.575812 (+1225421us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b540022f954. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.575923 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2048898944}
I20251023 02:05:31.577287 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:31.578187 22037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38998 (request call id 1532) took 1819 ms (client timeout 1605 ms). Trace:
I20251023 02:05:31.578678 22108 raft_consensus.cc:1240] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20251023 02:05:31.578588 22037 rpcz_store.cc:269] 1023 02:05:29.758988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:29.875142 (+116154us) service_pool.cc:225] Handling call
1023 02:05:31.555591 (+1680449us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003aa554. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.578123 (+ 22532us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2619262208}
I20251023 02:05:31.563959 24465 consensus_queue.cc:237] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.389216 24677 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.579761 22108 raft_consensus.cc:3055] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:31.587929 22108 raft_consensus.cc:740] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:31.574870 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:31.590875 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:31.591029 22105 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:31.563047 24808 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.592423 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:31.592578 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:31.592751 21957 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251023 02:05:31.593027 21961 raft_consensus.cc:2468] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:31.593457 24647 consensus_queue.cc:1059] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251023 02:05:31.593791 21958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1658) took 1477 ms. Trace:
I20251023 02:05:31.594277 24647 raft_consensus.cc:3055] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 LEADER]: Stepping down as leader of term 9
W20251023 02:05:31.594755 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 593610us
W20251023 02:05:31.595082 24781 log.cc:927] Time spent T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 1.058s user 0.009s sys 0.000s
I20251023 02:05:31.594964 24647 raft_consensus.cc:740] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:31.589908 22108 consensus_queue.cc:260] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.596084 22108 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.596279 24796 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:31.596097 24647 consensus_queue.cc:260] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.597487 24647 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:31.580855 24507 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:31.601074 24793 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.601562 24793 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.602665 22618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 278) took 3160 ms. Trace:
I20251023 02:05:31.582615 22035 raft_consensus.cc:2468] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
W20251023 02:05:31.583493 22002 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:31.603071 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:31.603035 22618 rpcz_store.cc:276] 1023 02:05:28.442770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:28.442946 (+ 176us) service_pool.cc:225] Handling call
1023 02:05:28.496903 (+ 53957us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:30.994097 (+2497194us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.602462 (+608365us) spinlock_profiling.cc:243] Waited 2.49 s on lock 0x7b5400600c54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:31.602607 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5735124224}
I20251023 02:05:31.594322 22076 leader_election.cc:400] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7.
I20251023 02:05:31.604429 21954 raft_consensus.cc:1240] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251023 02:05:31.604324 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:31.596814 24796 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:31.605211 21956 raft_consensus.cc:2393] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:31.594200 21958 rpcz_store.cc:276] 1023 02:05:30.116035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.529056 (+413021us) service_pool.cc:225] Handling call
1023 02:05:30.529277 (+ 221us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:30.529441 (+ 164us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:30.529688 (+ 247us) log.cc:844] Serialized 14 byte log entry
1023 02:05:31.593313 (+1063625us) log.cc:844] Serialized 14 byte log entry
1023 02:05:31.593457 (+ 144us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:31.593508 (+ 51us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:31.593548 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.593650 (+ 102us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:31.593742 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":583130,"prepare.queue_time_us":3002,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":83,"raft.queue_time_us":600045,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"thread_start_us":476965,"threads_started":1,"wal-append.queue_time_us":581993}
I20251023 02:05:31.635636 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.636438 22109 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:31.600505 21957 raft_consensus.cc:1240] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251023 02:05:31.604506 22076 leader_election.cc:403] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20251023 02:05:31.609018 21961 raft_consensus.cc:1240] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251023 02:05:31.610169 22623 raft_consensus.cc:1275] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251023 02:05:31.642530 22623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 386) took 3919 ms. Trace:
W20251023 02:05:31.609071 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 145446us
I20251023 02:05:31.643107 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:31.643872 22077 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:31.512810 21858 rpcz_store.cc:269] 1023 02:05:13.597868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:13.602234 (+ 4366us) service_pool.cc:225] Handling call
1023 02:05:13.794786 (+192552us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b540042f254. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa02fe0 00007f9c5faa1aca 00007f9c65813bbc 00007f9c6590de7e 00007f9c63e0d4a4 00007f9c63e0d432 00007f9c63e0d361 00007f9c63e0d2dc 00007f9c63e0c592 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:31.512340 (+17717554us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 02:05:26.190878 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 02:05:26.198063 (+ 7185us) write_op.cc:432] Acquiring schema lock in shared mode
1023 02:05:26.198124 (+ 61us) write_op.cc:435] Acquired schema lock
1023 02:05:26.198146 (+ 22us) tablet.cc:660] Decoding operations
1023 02:05:26.199131 (+ 985us) write_op.cc:620] Acquiring the partition lock for write op
1023 02:05:26.199210 (+ 79us) write_op.cc:641] Partition lock acquired for write op
1023 02:05:26.199237 (+ 27us) tablet.cc:683] Acquiring locks for 9 operations
1023 02:05:26.200131 (+ 894us) tablet.cc:699] Row locks acquired
1023 02:05:26.200158 (+ 27us) write_op.cc:260] PREPARE: finished
1023 02:05:26.200322 (+ 164us) write_op.cc:270] Start()
1023 02:05:26.200477 (+ 155us) write_op.cc:276] Timestamp: P: 1761185126200256 usec, L: 0
1023 02:05:26.200496 (+ 19us) op_driver.cc:348] REPLICATION: starting
1023 02:05:26.201163 (+ 667us) log.cc:844] Serialized 3912 byte log entry
1023 02:05:27.552204 (+1351041us) op_driver.cc:464] REPLICATION: finished
1023 02:05:27.866484 (+314280us) write_op.cc:301] APPLY: starting
1023 02:05:27.868886 (+ 2402us) tablet.cc:1365] starting BulkCheckPresence
1023 02:05:27.869151 (+ 265us) tablet.cc:1368] finished BulkCheckPresence
1023 02:05:27.874489 (+ 5338us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 02:05:28.113337 (+238848us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 02:05:28.113793 (+ 456us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
1023 02:05:28.113887 (+ 94us) write_op.cc:312] APPLY: finished
1023 02:05:28.114311 (+ 424us) log.cc:844] Serialized 71 byte log entry
1023 02:05:28.114550 (+ 239us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:05:28.114721 (+ 171us) write_op.cc:454] Released schema lock
1023 02:05:28.114808 (+ 87us) write_op.cc:341] FINISH: Updating metrics
1023 02:05:28.115236 (+ 428us) write_op.cc:489] Releasing partition, row and schema locks
1023 02:05:28.115265 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"dns_us":223,"mutex_wait_us":3690700,"reported_tablets":15,"spinlock_wait_cycles":9881344,"child_traces":[["op",{"apply.queue_time_us":252584,"apply.run_cpu_time_us":5024,"apply.run_wall_time_us":283310,"num_ops":9,"prepare.queue_time_us":297992,"prepare.run_cpu_time_us":23828,"prepare.run_wall_time_us":345956,"raft.queue_time_us":715247,"raft.run_cpu_time_us":17936,"raft.run_wall_time_us":184799,"replication_time_us":1351555,"thread_start_us":1228452,"threads_started":4,"wal-append.queue_time_us":528198,"wal-append.run_cpu_time_us":42682,"wal-append.run_wall_time_us":2468133}]]}
W20251023 02:05:31.645603 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 51385.9us
I20251023 02:05:31.646304 22030 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:31.644529 22000 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:31.646888 22030 raft_consensus.cc:606] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:31.647253 22030 raft_consensus.cc:614] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:31.647853 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:31.648375 21954 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.648699 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:31.649760 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:31.650275 21956 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:31.650578 22590 leader_election.cc:304] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:31.650998 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:31.651822 22028 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:31.652187 21956 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:31.642856 22623 rpcz_store.cc:276] 1023 02:05:27.723112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:27.726937 (+ 3825us) service_pool.cc:225] Handling call
1023 02:05:27.765677 (+ 38740us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:30.241384 (+2475707us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540052db20. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa0e96d 00007f9c5f9f50f4 00007f9c5fa77097 00007f9c5fa8fb87 00007f9c5fa9137e 00007f9c5fa9b6d9 00007f9c5fa9bfa6 00007f9c5fa9892f 00007f9c5fa983de 00007f9c64a419d9
1023 02:05:31.490426 (+1249042us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b580088e2d0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5f9f4ffa 00007f9c5fa77097 00007f9c5fa8fb87 00007f9c5fa9137e 00007f9c5fa9b6d9 00007f9c5fa9bfa6 00007f9c5fa9892f 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34
1023 02:05:31.642290 (+151864us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.642479 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4914994944}
I20251023 02:05:31.656742 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.560963 24735 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.657410 22035 raft_consensus.cc:2368] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 8. Current term is 9.
I20251023 02:05:31.627640 22105 raft_consensus.cc:1240] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9. Ops: [8.3-8.3]
W20251023 02:05:31.620962 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:31.659857 22029 raft_consensus.cc:1240] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:31.659911 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.660596 22034 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:31.667044 22590 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:31.669239 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.670226 22028 raft_consensus.cc:2393] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:31.671227 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.674295 21956 raft_consensus.cc:1240] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251023 02:05:31.587677 24686 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:31.589437 24552 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.683679 22029 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 8. Current term is 9. Ops: []
I20251023 02:05:31.587643 24714 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Leader election won for term 10
I20251023 02:05:31.684408 22110 raft_consensus.cc:2468] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:31.600461 24667 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20251023 02:05:31.709657 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 58317.8us
I20251023 02:05:31.632644 24295 raft_consensus.cc:2749] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:31.716415 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:31.718329 24832 log.cc:927] Time spent T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20251023 02:05:31.761762 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:31.719230 22106 raft_consensus.cc:2410] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251023 02:05:31.681890 24547 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.720340 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251023 02:05:31.574005 24744 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.646234 24796 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:31.721853 22620 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9. Ops: []
I20251023 02:05:31.722625 24808 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.606392 24346 consensus_queue.cc:1048] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.125s
I20251023 02:05:31.764668 24686 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.723507 24591 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:31.813966 24591 raft_consensus.cc:697] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:31.814949 24591 consensus_queue.cc:237] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.724146 24754 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.709395 24668 consensus_queue.cc:1059] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20251023 02:05:31.726539 22036 raft_consensus.cc:1240] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:31.726179 24552 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.727247 24714 raft_consensus.cc:697] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:31.727550 24735 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:31.595417 24577 consensus_queue.cc:1059] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.087s
I20251023 02:05:31.728403 24439 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.728832 24807 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:31.674295 24725 consensus_queue.cc:1048] T 2da2ca0f34944635a558b8678ecc206f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20251023 02:05:31.661536 24810 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.730144 24677 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.682391 24670 raft_consensus.cc:2804] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election won for term 7
W20251023 02:05:31.734122 24830 log.cc:927] Time spent T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
I20251023 02:05:31.672618 24798 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:31.735569 24655 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:31.737227 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:31.738111 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
W20251023 02:05:31.738668 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:31.717415 24822 consensus_queue.cc:1059] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20251023 02:05:31.742789 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:31.743466 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.744472 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:31.745095 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.751281 22624 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:31.751216 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.756328 22107 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 9.
I20251023 02:05:31.759572 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 216) took 1305 ms. Trace:
W20251023 02:05:31.760526 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 152005us
I20251023 02:05:31.766373 22028 raft_consensus.cc:2393] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:31.766558 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:05:31.721305 24793 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.771030 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.850633 24808 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.895665 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.750006 24610 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.780036 22102 raft_consensus.cc:1275] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20251023 02:05:31.779983 22109 raft_consensus.cc:2468] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:31.745739 24667 raft_consensus.cc:493] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:31.683383 24507 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
W20251023 02:05:31.789516 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 79345.5us
I20251023 02:05:31.915069 22102 pending_rounds.cc:85] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 3
I20251023 02:05:31.915378 22102 pending_rounds.cc:107] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251023 02:05:31.915616 22102 raft_consensus.cc:2889] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:31.916778 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.917330 22109 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:31.791363 22108 raft_consensus.cc:1275] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20251023 02:05:31.917725 22108 pending_rounds.cc:85] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 3
I20251023 02:05:31.917949 22108 pending_rounds.cc:107] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251023 02:05:31.918203 22108 raft_consensus.cc:2889] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:31.919270 22000 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:31.806773 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.920133 22106 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:31.826157 24457 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.812687 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1587) took 1351 ms. Trace:
I20251023 02:05:31.818397 24368 consensus_queue.cc:1048] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
W20251023 02:05:31.813288 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:31.819558 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 167020us
I20251023 02:05:31.821712 22033 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:31.831633 22104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1585) took 1382 ms. Trace:
I20251023 02:05:31.833693 21954 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:31.789806 24292 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.855381 22037 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:31.852059 24576 raft_consensus.cc:3055] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:31.870995 22111 raft_consensus.cc:2410] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
W20251023 02:05:31.894586 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:31.943678 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:31.867848 22077 leader_election.cc:400] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 8. Current term is 9.
I20251023 02:05:31.945806 22077 leader_election.cc:403] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
W20251023 02:05:31.896189 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:31.946882 22075 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:31.898568 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.902818 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:05:31.947856 22107 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:31.949687 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:31.903707 22034 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:31.907213 22028 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:31.952440 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:31.907871 21925 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:31.908592 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:31.866719 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.853094 22616 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20251023 02:05:31.970053 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 102342us
I20251023 02:05:31.900521 21955 rpcz_store.cc:276] 1023 02:05:30.454437 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.130460 (+676023us) service_pool.cc:225] Handling call
1023 02:05:31.130754 (+ 294us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:31.130949 (+ 195us) raft_consensus.cc:1534] Early marking committed up to index 6
1023 02:05:31.131224 (+ 275us) log.cc:844] Serialized 14 byte log entry
1023 02:05:31.296026 (+164802us) log.cc:844] Serialized 14 byte log entry
1023 02:05:31.759202 (+463176us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:31.759257 (+ 55us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:31.759305 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.759380 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:31.759509 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":462933,"prepare.queue_time_us":4755,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":4957,"raft.queue_time_us":463811,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":164345,"threads_started":1,"wal-append.queue_time_us":280174}
I20251023 02:05:31.970880 22106 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:31.971021 22077 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:31.919440 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.920331 22000 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:31.810070 24689 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.920607 22002 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:31.972180 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 64389.9us
W20251023 02:05:31.862253 24640 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.271s user 0.020s sys 0.038s
I20251023 02:05:31.779592 21908 placement_policy_util.cc:244] tablet 6dddaeefc3d449a0ba9b3e6695e37507: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
W20251023 02:05:31.973206 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 52951.1us
I20251023 02:05:31.973968 22108 raft_consensus.cc:2393] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 9: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:31.922833 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:31.976656 22624 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 9: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:31.927018 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.812556 24686 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.932297 24793 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.933130 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.933619 22104 rpcz_store.cc:276] 1023 02:05:30.449201 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.452120 (+ 2919us) service_pool.cc:225] Handling call
1023 02:05:30.510344 (+ 58224us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:31.189293 (+678949us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:31.189335 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:31.213003 (+ 23668us) log.cc:844] Serialized 26 byte log entry
1023 02:05:31.379389 (+166386us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:31.379426 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.379490 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:31.831316 (+451826us) raft_consensus.cc:1685] finished
1023 02:05:31.831357 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:31.831551 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":643408,"raft.queue_time_us":664412,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":3679,"spinlock_wait_cycles":67840,"thread_start_us":130547,"threads_started":2,"wal-append.queue_time_us":286098}
I20251023 02:05:31.864648 24670 raft_consensus.cc:697] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:31.937482 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:31.866369 24810 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.927661 22103 rpcz_store.cc:276] 1023 02:05:30.461639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:30.469126 (+ 7487us) service_pool.cc:225] Handling call
1023 02:05:30.469428 (+ 302us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:31.153866 (+684438us) raft_consensus.cc:1534] Early marking committed up to index 6
1023 02:05:31.176483 (+ 22617us) log.cc:844] Serialized 14 byte log entry
1023 02:05:31.289144 (+112661us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:31.295369 (+ 6225us) log.cc:844] Serialized 26 byte log entry
1023 02:05:31.295595 (+ 226us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:31.295652 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.295755 (+ 103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:31.336028 (+ 40273us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b18003430c0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9eb3d2 00007f9c5c9ed7a1 00007f9c5c9ed759 00007f9c5c9ed6e9 00007f9c5c9ed6b1 00007f9c5c9ec9ad 0000000000379254 0000000000379099 00007f9c5c9ce75a
1023 02:05:31.812318 (+476290us) raft_consensus.cc:1685] finished
1023 02:05:31.812450 (+ 132us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:31.812615 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":676193,"prepare.queue_time_us":16645,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":696406,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":17008,"spinlock_wait_cycles":3907968,"thread_start_us":87599,"threads_started":1,"wal-append.queue_time_us":398646}
I20251023 02:05:31.939495 22623 raft_consensus.cc:1240] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
W20251023 02:05:31.937803 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:31.941228 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1694) took 1110 ms. Trace:
I20251023 02:05:31.989126 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:31.924819 24507 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:31.864903 24691 consensus_queue.cc:1048] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20251023 02:05:31.942088 24292 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.942564 24744 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:31.939234 21928 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:31.956349 22036 raft_consensus.cc:2393] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 8: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:31.918407 24439 consensus_queue.cc:1059] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.059s
I20251023 02:05:31.928560 24667 raft_consensus.cc:3060] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:31.932963 24813 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.943254 24576 raft_consensus.cc:740] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:31.958307 24831 consensus_queue.cc:1059] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251023 02:05:31.961846 24798 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:31.964939 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:31.965169 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:32.023936 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:32.024686 22103 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:32.024973 22107 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:31.906898 24807 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:31.967893 21958 raft_consensus.cc:1275] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251023 02:05:31.992311 24577 raft_consensus.cc:2804] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:31.968912 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:32.030861 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 91726.6us
I20251023 02:05:32.031265 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:31.969280 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:32.031610 21928 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:32.032135 21959 raft_consensus.cc:2468] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:31.957211 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.929831 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:31.931164 24760 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.936633 24346 raft_consensus.cc:2804] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:31.989490 22102 rpcz_store.cc:276] 1023 02:05:30.830468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.588603 (+758135us) service_pool.cc:225] Handling call
1023 02:05:31.779610 (+191007us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:31.916013 (+136403us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:31.941043 (+ 25030us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54000aaad0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:31.941170 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":438611840}
I20251023 02:05:31.990213 22032 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:31.991425 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:31.957412 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.038709 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:32.039983 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:31.982666 21925 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:31.926627 24610 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:31.980115 22618 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
W20251023 02:05:31.981417 24640 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.393s user 0.020s sys 0.038s
I20251023 02:05:32.012415 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:31.920622 24808 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.936368 24725 consensus_queue.cc:1059] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 6.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20251023 02:05:32.010252 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:32.010231 22587 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:31.934643 24748 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:31.865284 24714 consensus_queue.cc:237] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:32.021701 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:31.983922 24634 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:31.987864 24689 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:31.971732 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:31.833051 24754 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:32.075215 24689 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } }
I20251023 02:05:31.943801 24677 consensus_queue.cc:1059] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.067s
I20251023 02:05:32.075981 24748 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:32.049887 24346 raft_consensus.cc:697] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:32.076565 24677 raft_consensus.cc:3055] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:32.077021 24677 raft_consensus.cc:740] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
W20251023 02:05:32.077517 24833 log.cc:927] Time spent T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20251023 02:05:32.077389 24346 consensus_queue.cc:237] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:32.077904 24677 consensus_queue.cc:260] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:32.078895 24668 raft_consensus.cc:3055] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:32.079128 24677 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:31.992725 24507 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:31.968537 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:32.080672 22108 raft_consensus.cc:1240] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:31.980753 24822 raft_consensus.cc:2749] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 8. Current term is 9.
I20251023 02:05:32.034215 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:32.089275 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:32.089900 22034 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:32.033372 22030 raft_consensus.cc:2393] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 6: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:32.046269 24823 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:32.047688 22028 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20251023 02:05:32.026706 24744 raft_consensus.cc:2804] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election won for term 9
I20251023 02:05:32.027817 24798 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:32.051939 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:32.102393 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:32.103132 21956 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:32.052996 24610 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:32.052764 24576 consensus_queue.cc:260] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:32.053874 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:32.107649 22102 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:32.054350 22033 raft_consensus.cc:1275] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251023 02:05:32.055485 24808 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:32.114104 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:31.991648 24793 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:32.056252 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:32.033986 24577 raft_consensus.cc:493] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:32.065708 24725 consensus_queue.cc:1048] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:32.066259 24655 consensus_queue.cc:1059] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:32.067829 22109 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:32.068713 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:32.069908 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:32.112274 24823 raft_consensus.cc:697] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:32.144505 21955 raft_consensus.cc:2393] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 5: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:32.071724 24640 raft_consensus.cc:493] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:32.074352 22616 raft_consensus.cc:1240] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20251023 02:05:32.079177 22111 raft_consensus.cc:1275] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20251023 02:05:32.146118 22111 pending_rounds.cc:85] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 6
I20251023 02:05:32.146467 22111 pending_rounds.cc:107] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20251023 02:05:32.146708 22111 raft_consensus.cc:2889] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:32.032750 21928 leader_election.cc:400] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 8. Current term is 9.
I20251023 02:05:32.147686 21928 leader_election.cc:403] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20251023 02:05:32.083252 24762 consensus_queue.cc:1059] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20251023 02:05:32.083777 22108 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 9. Current term is 10. Ops: []
I20251023 02:05:32.155354 22108 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 8. Current term is 9. Ops: []
W20251023 02:05:32.084699 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 102160us
I20251023 02:05:32.157549 21925 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:32.086421 22103 raft_consensus.cc:2468] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 4.
I20251023 02:05:32.027333 21859 catalog_manager.cc:5649] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 4 to 6, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1), no longer has a pending config: VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1), VOTER 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2), VOTER ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3), VOTER 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4). New cstate: current_term: 6 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: NON_VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20251023 02:05:32.050048 24701 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:32.096194 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:32.096439 22037 raft_consensus.cc:1240] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251023 02:05:32.047377 24789 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:32.110137 22034 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:32.061906 24647 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Leader election won for term 9
I20251023 02:05:32.062192 24831 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:32.025652 24813 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:32.034848 24807 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:32.111678 22001 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:32.175673 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 64110.6us
I20251023 02:05:32.148916 24576 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:32.041360 24760 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:32.077390 24689 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:32.087051 24668 raft_consensus.cc:740] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:32.095145 24514 raft_consensus.cc:3055] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Stepping down as leader of term 4
I20251023 02:05:32.095476 24439 raft_consensus.cc:2749] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251023 02:05:32.135645 22102 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 9.
I20251023 02:05:32.192425 22035 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:32.192739 22102 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251023 02:05:32.163793 24789 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:32.148268 21957 raft_consensus.cc:1275] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20251023 02:05:32.148368 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 115750us
I20251023 02:05:32.194861 22109 raft_consensus.cc:1240] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20251023 02:05:32.181219 22076 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:32.082502 22104 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:32.162101 22111 raft_consensus.cc:1240] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:32.199530 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.200510 22034 raft_consensus.cc:2393] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 9: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:32.162580 22103 raft_consensus.cc:1275] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20251023 02:05:32.162808 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 186613us
I20251023 02:05:32.164352 22587 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:32.173979 24457 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:32.139324 24808 raft_consensus.cc:3055] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Stepping down as leader of term 8
I20251023 02:05:32.138666 24577 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:32.144392 24655 consensus_queue.cc:1243] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer 818907d63f1f4d84ba267ebe6ee837e0 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20251023 02:05:32.152482 21956 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:32.063028 24634 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:32.179425 21859 catalog_manager.cc:5649] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 4 to 5, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 5 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:32.174870 24779 consensus_queue.cc:1048] T 774c010489aa4775bb6d636926900529 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20251023 02:05:32.107064 24670 consensus_queue.cc:237] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:32.169834 24624 raft_consensus.cc:2804] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:32.191387 24514 raft_consensus.cc:740] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:32.209663 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 197567us
I20251023 02:05:32.209914 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.212405 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.258831 22077 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:32.212512 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:32.213238 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:32.214113 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 267371us
I20251023 02:05:32.260085 22077 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:32.218941 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:32.225431 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 61170.8us
I20251023 02:05:32.162916 24611 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.027s
I20251023 02:05:32.228051 24744 raft_consensus.cc:697] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:32.161746 24640 raft_consensus.cc:515] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:32.183769 24647 raft_consensus.cc:697] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:32.233538 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:32.279102 22102 raft_consensus.cc:2410] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20251023 02:05:32.234376 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:32.235845 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:32.281837 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:32.236940 22000 leader_election.cc:304] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:32.237989 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:32.283440 21957 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:32.241950 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:32.242903 21958 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:32.233305 24691 consensus_queue.cc:1059] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.007s
W20251023 02:05:32.183800 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 263275us
I20251023 02:05:32.210723 24292 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:32.165004 21954 raft_consensus.cc:2393] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 8: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:32.209427 24686 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:32.259811 22034 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:32.408299 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.261996 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:32.276152 24851 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:32.260120 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.223289 24789 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:32.184926 24748 consensus_queue.cc:1059] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.056s
I20251023 02:05:32.267110 21925 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:32.425860 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 158840us
I20251023 02:05:32.267460 22077 leader_election.cc:304] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:32.258778 22037 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:32.226531 24552 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:32.225225 24808 raft_consensus.cc:740] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:32.180472 24791 consensus_queue.cc:1059] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20251023 02:05:32.270746 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.269930 24668 consensus_queue.cc:260] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.2, Last appended by leader: 2, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:32.241355 24591 raft_consensus.cc:2804] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 FOLLOWER]: Leader election won for term 4
I20251023 02:05:32.246879 24634 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:32.194245 24762 consensus_queue.cc:1059] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20251023 02:05:32.247187 24346 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:32.366847 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:32.539510 21956 raft_consensus.cc:2368] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 9. Current term is 10.
I20251023 02:05:32.540386 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:32.540982 21956 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:32.252346 24847 raft_consensus.cc:3055] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Stepping down as leader of term 8
W20251023 02:05:32.368505 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 131645us
I20251023 02:05:32.366763 22035 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:32.371034 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:32.546940 22102 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:32.262718 24624 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:32.271344 24655 raft_consensus.cc:2749] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:32.291265 24547 raft_consensus.cc:2749] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7.
I20251023 02:05:32.276981 24822 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:32.390096 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:32.391575 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.240674 24836 raft_consensus.cc:2749] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 8. Current term is 9.
I20251023 02:05:32.240794 24806 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:32.409644 22103 raft_consensus.cc:2393] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 8: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:32.252231 21859 catalog_manager.cc:5649] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 5 to 6, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 6 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:32.275995 24465 consensus_queue.cc:1048] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.117s
I20251023 02:05:32.481158 22032 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:32.453400 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.471240 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:32.472422 21954 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:32.236940 24823 consensus_queue.cc:237] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:32.381188 24849 raft_consensus.cc:2804] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election won for term 8
W20251023 02:05:32.505617 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 238199us
W20251023 02:05:32.481454 24796 consensus_meta.cc:287] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.706s user 0.011s sys 0.000s
I20251023 02:05:32.531759 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:32.604394 22077 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:32.493474 24663 raft_consensus.cc:2804] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:32.547772 21956 raft_consensus.cc:1275] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20251023 02:05:32.516160 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:32.512215 22028 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:32.615689 21956 raft_consensus.cc:1240] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: [6.2-6.2]
I20251023 02:05:32.615795 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:32.616707 22028 raft_consensus.cc:2393] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 5: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
W20251023 02:05:32.554567 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:32.568673 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:32.581779 24624 raft_consensus.cc:3060] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:32.480363 24686 consensus_queue.cc:1048] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20251023 02:05:32.574016 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:32.627560 22102 raft_consensus.cc:2393] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 5: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:32.510356 24793 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:32.591712 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:32.628306 22034 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:32.494554 24744 consensus_queue.cc:237] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:32.639182 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:32.640622 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:32.599134 21954 raft_consensus.cc:1275] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251023 02:05:32.513437 21958 raft_consensus.cc:1275] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20251023 02:05:32.605044 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:32.641290 22102 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:32.647153 22032 raft_consensus.cc:3060] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:32.648742 22102 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 8. Current term is 9. Ops: []
I20251023 02:05:32.654914 22110 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
W20251023 02:05:32.627341 24844 log.cc:927] Time spent T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.404s user 0.001s sys 0.000s
I20251023 02:05:32.624056 24748 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:32.560002 24668 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:32.545953 24847 raft_consensus.cc:740] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:32.613561 21859 catalog_manager.cc:5649] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 3 to 5, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 5 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
W20251023 02:05:32.620769 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:32.570667 21955 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:32.572306 24807 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:32.558624 24591 raft_consensus.cc:697] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:32.570947 24346 raft_consensus.cc:697] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:32.623490 22103 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251023 02:05:32.727124 22103 raft_consensus.cc:1275] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251023 02:05:32.631450 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.597743 22030 raft_consensus.cc:2393] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 10: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
W20251023 02:05:32.634145 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 180480us
I20251023 02:05:32.620193 24547 raft_consensus.cc:3055] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:32.608606 22077 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:32.626858 24663 raft_consensus.cc:697] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:32.666574 24846 log.cc:927] Time spent T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.200s user 0.000s sys 0.000s
I20251023 02:05:32.605994 24806 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:32.613026 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:32.673478 22032 raft_consensus.cc:2468] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:32.762147 22033 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:32.679057 21957 raft_consensus.cc:2410] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
W20251023 02:05:32.700369 22104 consensus_meta.cc:287] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.502s user 0.006s sys 0.003s
I20251023 02:05:32.762888 22104 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 10.
I20251023 02:05:32.716857 22034 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:32.722559 21859 catalog_manager.cc:5649] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 4 to 5, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 5 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:05:32.634605 24691 consensus_queue.cc:1243] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251023 02:05:32.728199 22037 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:32.729913 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.588662 24647 consensus_queue.cc:237] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:32.667150 24840 log.cc:927] Time spent T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.295s user 0.000s sys 0.001s
W20251023 02:05:32.753317 24841 log.cc:927] Time spent T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.389s user 0.000s sys 0.001s
I20251023 02:05:32.750936 22618 raft_consensus.cc:1275] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20251023 02:05:32.619020 24796 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:32.725598 24748 raft_consensus.cc:697] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:32.788224 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:32.648603 24760 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:32.693634 24558 raft_consensus.cc:3055] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:32.811426 22618 pending_rounds.cc:85] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting all ops after (but not including) 5
I20251023 02:05:32.862902 22618 pending_rounds.cc:107] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20251023 02:05:32.863241 22618 raft_consensus.cc:2889] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:32.830283 22102 raft_consensus.cc:1275] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20251023 02:05:32.865890 22102 pending_rounds.cc:85] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting all ops after (but not including) 4
I20251023 02:05:32.866200 22102 pending_rounds.cc:107] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20251023 02:05:32.866484 22102 raft_consensus.cc:2889] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:32.777354 21859 catalog_manager.cc:5649] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 5 to 6, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 6 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:32.831137 22037 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 8. Current term is 9. Ops: []
W20251023 02:05:32.830976 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 222474us
I20251023 02:05:32.734707 24547 raft_consensus.cc:740] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:32.868341 22077 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:32.754524 24640 leader_election.cc:290] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:32.614877 24808 consensus_queue.cc:260] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:32.891152 22590 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:32.868276 21859 catalog_manager.cc:5649] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 4 to 5, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 5 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:32.897338 22077 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:32.898772 22077 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:32.915407 21961 raft_consensus.cc:2368] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7.
I20251023 02:05:32.773816 24465 consensus_queue.cc:1048] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.120s
I20251023 02:05:32.861968 24691 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:32.937783 21953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1313) took 1332 ms. Trace:
I20251023 02:05:32.923756 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.939033 22037 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:32.938323 21953 rpcz_store.cc:276] 1023 02:05:31.605854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.605963 (+ 109us) service_pool.cc:225] Handling call
1023 02:05:31.682284 (+ 76321us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:31.786269 (+103985us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:31.786301 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:31.788947 (+ 2646us) log.cc:844] Serialized 26 byte log entry
1023 02:05:32.017998 (+229051us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:32.018049 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:32.018129 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:32.937478 (+919349us) raft_consensus.cc:1685] finished
1023 02:05:32.937506 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:32.937709 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":103606,"raft.queue_time_us":106716,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":26510,"thread_start_us":171919,"threads_started":1,"wal-append.queue_time_us":317667}
I20251023 02:05:32.940403 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e32bb6c0622f44f8806f77eee0691856" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.924933 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:32.925923 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:32.967531 22032 raft_consensus.cc:2410] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251023 02:05:32.968936 22032 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 8. Current term is 9. Ops: []
I20251023 02:05:32.926949 21925 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:32.970196 24640 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:32.970713 24640 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:32.971139 24640 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:32.930032 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1314) took 1323 ms. Trace:
W20251023 02:05:32.933866 22002 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s user 0.010s sys 0.014s
W20251023 02:05:32.973929 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 565618us
I20251023 02:05:32.860242 24850 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:32.974869 24850 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:32.975800 24791 consensus_queue.cc:1048] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:32.976946 24850 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:32.973742 21960 rpcz_store.cc:276] 1023 02:05:31.606211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.606337 (+ 126us) service_pool.cc:225] Handling call
1023 02:05:31.673141 (+ 66804us) raft_consensus.cc:1471] Updating replica for 3 ops
1023 02:05:31.790358 (+117217us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:31.790410 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 3 ops
1023 02:05:31.838665 (+ 48255us) log.cc:844] Serialized 78 byte log entry
1023 02:05:31.921660 (+ 82995us) spinlock_profiling.cc:243] Waited 9.97 ms on lock 0x7b18002f2f00. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9cdd37 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c5f62f2bb 00007f9c5f634dc3 00007f9c5f635171 00007f9c5fa5465f
1023 02:05:32.192613 (+270953us) raft_consensus.cc:1645] Marking committed up to 8
1023 02:05:32.240364 (+ 47751us) log.cc:844] Serialized 14 byte log entry
1023 02:05:32.276790 (+ 36426us) log.cc:844] Serialized 14 byte log entry
1023 02:05:32.379486 (+102696us) log.cc:844] Serialized 14 byte log entry
1023 02:05:32.506156 (+126670us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:32.535197 (+ 29041us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:32.929790 (+394593us) raft_consensus.cc:1685] finished
1023 02:05:32.929816 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:32.929970 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":251025,"prepare.queue_time_us":54047,"prepare.run_wall_time_us":48,"raft.queue_time_us":378723,"raft.run_cpu_time_us":461,"raft.run_wall_time_us":61003,"spinlock_wait_cycles":225385216,"thread_start_us":388953,"threads_started":2,"wal-append.queue_time_us":430307}
I20251023 02:05:32.979594 24868 consensus_queue.cc:1048] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:05:32.977149 24791 consensus_queue.cc:1059] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251023 02:05:32.982164 24791 raft_consensus.cc:3055] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Stepping down as leader of term 8
I20251023 02:05:32.982625 24791 raft_consensus.cc:740] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:32.983830 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:32.861640 24558 raft_consensus.cc:740] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:32.860548 24813 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:33.018647 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 44083.3us
I20251023 02:05:33.018994 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1310) took 1417 ms. Trace:
I20251023 02:05:32.992877 22033 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:33.023689 21962 rpcz_store.cc:276] 1023 02:05:31.601404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.601513 (+ 109us) service_pool.cc:225] Handling call
1023 02:05:31.601814 (+ 301us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:31.602122 (+ 308us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:31.602161 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:31.602594 (+ 433us) log.cc:844] Serialized 26 byte log entry
1023 02:05:32.011187 (+408593us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:32.011226 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:32.011301 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:33.018762 (+1007461us) raft_consensus.cc:1685] finished
1023 02:05:33.018790 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:33.018935 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":445,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"thread_start_us":408215,"threads_started":1,"wal-append.queue_time_us":607497}
I20251023 02:05:32.845355 24793 consensus_queue.cc:1059] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.090s
W20251023 02:05:33.026867 24873 log.cc:927] Time spent T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
I20251023 02:05:33.028163 24748 consensus_queue.cc:237] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.030282 24754 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:32.891661 24655 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.126s
I20251023 02:05:33.030774 24806 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:33.031731 24655 raft_consensus.cc:2749] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:32.920876 24591 consensus_queue.cc:237] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.038726 24796 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:32.935451 24346 consensus_queue.cc:237] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:33.044454 24855 log.cc:927] Time spent T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.422s user 0.001s sys 0.000s
W20251023 02:05:33.045694 24856 log.cc:927] Time spent T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.455s user 0.000s sys 0.001s
I20251023 02:05:32.895071 24847 consensus_queue.cc:260] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.3, Last appended by leader: 3, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:32.977495 24850 raft_consensus.cc:2804] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:33.046468 24850 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:33.046845 24850 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:33.046013 21859 catalog_manager.cc:5649] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 2 to 4, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 4 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:33.048063 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1708) took 1928 ms. Trace:
I20251023 02:05:33.048251 21859 catalog_manager.cc:5649] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 3 to 4, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 4 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:32.916862 24864 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:33.050364 24861 raft_consensus.cc:493] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:33.018282 24691 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:33.046958 24852 log.cc:927] Time spent T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.313s user 0.000s sys 0.000s
I20251023 02:05:32.915424 24627 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.146s
I20251023 02:05:33.048460 22105 rpcz_store.cc:276] 1023 02:05:31.119872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.662013 (+542141us) service_pool.cc:225] Handling call
1023 02:05:31.724763 (+ 62750us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:31.725148 (+ 385us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:31.725176 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:31.725661 (+ 485us) log.cc:844] Serialized 26 byte log entry
1023 02:05:32.100102 (+374441us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:32.230171 (+130069us) log.cc:844] Serialized 14 byte log entry
1023 02:05:32.230371 (+ 200us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:32.230438 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:33.047857 (+817419us) raft_consensus.cc:1685] finished
1023 02:05:33.047880 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:33.048009 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7954,"prepare.queue_time_us":487634,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":10188,"raft.queue_time_us":15199,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":4957,"thread_start_us":503145,"threads_started":2,"wal-append.queue_time_us":500241}
I20251023 02:05:33.062981 22002 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:33.065516 24662 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.173861 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1594) took 1490 ms. Trace:
I20251023 02:05:33.177129 24668 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251023 02:05:33.091776 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:33.023089 24457 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.152741 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:33.179908 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1329) took 1314 ms. Trace:
W20251023 02:05:33.180341 24701 consensus_meta.cc:287] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.011s user 0.015s sys 0.001s
W20251023 02:05:33.068496 24667 consensus_meta.cc:287] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.043s user 0.011s sys 0.000s
I20251023 02:05:33.180634 24543 raft_consensus.cc:3055] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:33.181078 24851 raft_consensus.cc:3055] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:33.050889 24861 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.072068 21859 catalog_manager.cc:5649] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 2 to 5, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 5 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:33.180724 24850 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:33.181540 24576 consensus_meta.cc:287] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.997s user 0.013s sys 0.003s
W20251023 02:05:33.182760 24634 consensus_meta.cc:287] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.626s user 0.007s sys 0.006s
I20251023 02:05:33.050385 24864 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:33.184458 21955 raft_consensus.cc:2410] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 9 because replica has last-logged OpId of term: 8 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 2.
I20251023 02:05:33.015049 24808 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:33.023244 24807 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:33.065055 24845 log.cc:927] Time spent T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.495s user 0.001s sys 0.000s
W20251023 02:05:33.220842 22036 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.005s sys 0.003s
W20251023 02:05:33.088538 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:33.016125 24848 consensus_queue.cc:1175] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "02c53ba06cc3445a8534a84aa76b1c14" responder_term: 8 status { last_received { term: 8 index: 7 } last_committed_idx: 6 last_received_current_leader { term: 8 index: 7 } }
W20251023 02:05:33.235212 22000 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.003s sys 0.007s
W20251023 02:05:33.237371 22076 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:33.172886 24849 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.032s sys 0.061s
I20251023 02:05:33.136857 24624 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.063882 24547 consensus_queue.cc:260] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.136142 24663 consensus_queue.cc:237] T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.240196 22111 rpcz_store.cc:276] 1023 02:05:31.683030 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.152971 (+469941us) service_pool.cc:225] Handling call
1023 02:05:32.158538 (+ 5567us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:32.210295 (+ 51757us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:33.173620 (+963325us) spinlock_profiling.cc:243] Waited 5.21 ms on lock 0x7b54000fb250. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:33.173793 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11976960}
W20251023 02:05:33.541095 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:33.539914 22031 rpcz_store.cc:276] 1023 02:05:31.865653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.055075 (+189422us) service_pool.cc:225] Handling call
1023 02:05:32.055415 (+ 340us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:32.490566 (+435151us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:32.490608 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:32.493007 (+ 2399us) log.cc:844] Serialized 52 byte log entry
1023 02:05:32.580521 (+ 87514us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:33.175933 (+595412us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b48001701d8. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c649a7971 00007f9c6499ea37 00007f9c6499e8ca 00007f9c64a95a0f 00007f9c64a956d7 00007f9c64aa9cfc 00007f9c64aa9ca8 00007f9c64aa9c28 00007f9c64aa9be3 00007f9c64aa8e8c 00007f9c639e5326
1023 02:05:33.178913 (+ 2980us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b18002bf000. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5f63d55c 00007f9c5c9ce3bb 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9e8445 00007f9c63488b27 00007f9c5faa3537 00007f9c5fab2c4f
1023 02:05:33.179056 (+ 143us) log.cc:844] Serialized 14 byte log entry
1023 02:05:33.179446 (+ 390us) log.cc:844] Serialized 14 byte log entry
1023 02:05:33.179616 (+ 170us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:33.179686 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:33.179724 (+ 38us) raft_consensus.cc:1685] finished
1023 02:05:33.179734 (+ 10us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:33.179855 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25400,"prepare.queue_time_us":42454,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":14149,"raft.queue_time_us":179056,"raft.run_cpu_time_us":3244,"raft.run_wall_time_us":936014,"spinlock_wait_cycles":64555008,"thread_start_us":547779,"threads_started":3,"wal-append.queue_time_us":167377}
I20251023 02:05:33.253036 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:33.507104 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 1.32891e+06us
W20251023 02:05:33.543900 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:33.544179 22076 leader_election.cc:400] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 9. Current term is 10.
I20251023 02:05:33.544503 24694 raft_consensus.cc:2749] T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:33.374537 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:33.546126 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 483264us
I20251023 02:05:33.546772 24624 leader_election.cc:290] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:33.233201 24577 consensus_meta.cc:287] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.980s user 0.010s sys 0.002s
I20251023 02:05:33.547549 24577 raft_consensus.cc:515] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.548419 21928 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:33.548509 24634 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.544787 24823 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:33.224371 24852 log.cc:927] Time spent T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20251023 02:05:33.254047 24558 consensus_queue.cc:260] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.313020 24872 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.549679 24871 raft_consensus.cc:493] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:33.549809 24667 raft_consensus.cc:515] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.550503 24701 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:33.550647 21928 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:33.553682 22000 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.851s user 0.014s sys 0.009s
I20251023 02:05:33.310444 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251023 02:05:33.546936 24849 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.879s user 0.036s sys 0.071s
I20251023 02:05:33.222028 24760 consensus_queue.cc:1048] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.359s
W20251023 02:05:33.558301 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:33.624665 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:33.625504 21962 raft_consensus.cc:1240] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:33.544577 22076 leader_election.cc:403] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
W20251023 02:05:33.627442 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 83398.4us
I20251023 02:05:33.557919 24851 raft_consensus.cc:740] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:33.547567 22105 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 6.
I20251023 02:05:33.541134 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:33.541503 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1323) took 1325 ms. Trace:
I20251023 02:05:33.575342 24578 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251023 02:05:33.581661 22102 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251023 02:05:33.551021 24543 raft_consensus.cc:740] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:33.560832 21859 catalog_manager.cc:5649] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 3 to 4, leader changed from 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 4 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20251023 02:05:33.584689 22077 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.020s sys 0.032s
W20251023 02:05:33.636631 22077 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.737s user 0.020s sys 0.032s
W20251023 02:05:33.637194 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 738575us
W20251023 02:05:33.637491 22106 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.143s user 0.010s sys 0.044s
I20251023 02:05:33.638985 22077 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:33.577561 24754 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.639271 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:33.640846 22077 leader_election.cc:304] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:33.641176 22102 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:33.640147 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 89598.2us
I20251023 02:05:33.643266 21928 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:33.638514 22106 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.144s user 0.011s sys 0.044s
I20251023 02:05:33.617281 24871 raft_consensus.cc:515] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.280710 24655 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:33.618353 24849 raft_consensus.cc:697] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:33.560176 22616 raft_consensus.cc:1240] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [term 4 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
W20251023 02:05:33.622195 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 75388.1us
I20251023 02:05:33.623102 22031 raft_consensus.cc:1275] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251023 02:05:33.712416 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1563) took 1093 ms. Trace:
I20251023 02:05:33.712896 22031 rpcz_store.cc:276] 1023 02:05:32.619417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:33.575663 (+956246us) service_pool.cc:225] Handling call
1023 02:05:33.622687 (+ 47024us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:33.712082 (+ 89395us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:33.712339 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:33.558754 22036 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.092s user 0.009s sys 0.009s
I20251023 02:05:33.559945 24864 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:33.539680 22037 raft_consensus.cc:1275] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251023 02:05:33.575407 24880 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:33.630789 22111 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:33.630787 21955 rpcz_store.cc:276] 1023 02:05:32.215930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.292409 (+ 76479us) service_pool.cc:225] Handling call
1023 02:05:33.541379 (+1248970us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:33.615077 24507 consensus_meta.cc:287] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 1.535s user 0.008s sys 0.004s
I20251023 02:05:33.720695 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:33.647359 24748 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 9. Current term is 10.
I20251023 02:05:33.651026 21959 raft_consensus.cc:2393] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 9: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:33.620951 24547 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:33.735553 24822 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:33.736027 24822 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:33.650208 21859 catalog_manager.cc:5649] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
W20251023 02:05:33.702216 24640 consensus_meta.cc:287] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.731s user 0.012s sys 0.006s
I20251023 02:05:33.619819 22103 raft_consensus.cc:1240] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
W20251023 02:05:33.688596 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 45445.6us
I20251023 02:05:33.629437 24576 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
W20251023 02:05:33.644918 24670 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.034s sys 0.042s
I20251023 02:05:33.712286 22002 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:33.561873 24624 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:33.649327 24691 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:33.577477 24457 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.719424 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:33.769085 21953 raft_consensus.cc:2410] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251023 02:05:33.616386 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:33.574910 24807 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.722270 21925 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:33.635288 24668 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.719532 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1562) took 1137 ms. Trace:
I20251023 02:05:33.728746 22111 raft_consensus.cc:1240] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
W20251023 02:05:33.688395 24793 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.014s sys 0.023s
I20251023 02:05:33.703425 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:33.742172 21961 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1320) took 2134 ms (client timeout 1816 ms). Trace:
I20251023 02:05:33.580632 24823 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.746886 21928 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:33.691951 24876 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.585819 24872 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:33.792654 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 45895.1us
I20251023 02:05:33.694983 24701 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251023 02:05:33.575330 24662 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.710279 24806 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251023 02:05:33.635882 24295 consensus_queue.cc:1048] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.097s
I20251023 02:05:33.765118 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:33.766391 24790 raft_consensus.cc:3055] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:33.766729 24624 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.756855 24576 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.739367 24507 raft_consensus.cc:515] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.635103 24813 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:33.791139 24578 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:33.733904 24677 consensus_meta.cc:287] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 1.654s user 0.014s sys 0.006s
W20251023 02:05:33.772325 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 60175.5us
I20251023 02:05:33.830957 22002 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:33.831933 22589 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:33.833093 22002 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:33.772931 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 50782.2us
I20251023 02:05:33.834806 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:33.753592 24670 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s user 0.035s sys 0.043s
I20251023 02:05:33.776021 22037 rpcz_store.cc:276] 1023 02:05:32.582299 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:33.252655 (+670356us) service_pool.cc:225] Handling call
1023 02:05:33.539307 (+286652us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:33.719282 (+179975us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:33.719479 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:33.742889 24694 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:33.737505 24692 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:33.782855 21961 rpcz_store.cc:269] 1023 02:05:31.607688 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.642807 (+ 35119us) service_pool.cc:225] Handling call
1023 02:05:33.741918 (+2099111us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54000bf9d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:33.742103 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2924269824}
I20251023 02:05:33.780918 22624 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251023 02:05:33.788395 22102 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:33.705045 24754 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.702642 24627 consensus_queue.cc:1059] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20251023 02:05:33.794198 21928 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:33.794202 22111 raft_consensus.cc:1240] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
W20251023 02:05:33.787403 24789 consensus_meta.cc:287] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 1.307s user 0.014s sys 0.000s
I20251023 02:05:33.757323 24275 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.766606 24368 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:33.782668 24793 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s user 0.014s sys 0.023s
W20251023 02:05:33.790172 24798 consensus_meta.cc:287] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 1.673s user 0.007s sys 0.010s
W20251023 02:05:33.814507 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 173752us
I20251023 02:05:33.787052 24744 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:33.766469 21859 catalog_manager.cc:5649] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:33.788723 24879 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.754516 24640 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:33.738024 24860 log.cc:927] Time spent T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.244s user 0.000s sys 0.000s
I20251023 02:05:33.827389 22105 raft_consensus.cc:1240] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:33.828697 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:33.823035 24885 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.787530 24876 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.779371 24668 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.928720 24668 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:33.836864 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:33.844200 22002 leader_election.cc:304] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:33.843009 24813 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.808202 24790 raft_consensus.cc:740] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
W20251023 02:05:33.843734 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 208500us
I20251023 02:05:33.846880 22102 raft_consensus.cc:1275] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251023 02:05:33.759375 24543 consensus_queue.cc:260] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.789647 24691 consensus_queue.cc:1059] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20251023 02:05:33.806064 24897 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:33.758385 24851 consensus_queue.cc:260] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.864038 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74071b399ecc41808f0ac84dc836af4c" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:33.821842 24806 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251023 02:05:33.877542 22111 raft_consensus.cc:1240] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20251023 02:05:33.878690 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 84613.3us
I20251023 02:05:33.825614 24346 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.798659 24822 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:33.837436 24762 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.000s sys 0.040s
I20251023 02:05:33.841800 24689 raft_consensus.cc:2804] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:33.886652 24789 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.889205 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:33.844549 24670 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.886027 22617 raft_consensus.cc:1240] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:33.885942 24740 raft_consensus.cc:2749] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:33.888201 24667 leader_election.cc:290] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:33.885821 24275 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.891537 24457 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:33.837073 24889 consensus_queue.cc:1048] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.067s
I20251023 02:05:33.862380 24892 raft_consensus.cc:2804] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:33.860616 24714 raft_consensus.cc:2749] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251023 02:05:33.861155 24677 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.864104 24634 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:33.874051 24850 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:33.879849 24853 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.839493 24578 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:33.842689 24692 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.880486 24465 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20251023 02:05:33.816150 24849 consensus_queue.cc:237] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:33.942309 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:34.318784 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:34.052379 21960 raft_consensus.cc:2393] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:05:33.978134 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:34.342753 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:33.969410 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 135263us
I20251023 02:05:34.343545 22037 raft_consensus.cc:3060] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:34.343292 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:34.345345 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:33.938462 21859 catalog_manager.cc:5649] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 1 to 6, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 6 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
W20251023 02:05:34.341825 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 510094us
I20251023 02:05:34.348409 21928 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:34.034695 22110 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39646 (request call id 1596) took 2351 ms (client timeout 1743 ms). Trace:
I20251023 02:05:34.348739 22589 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:34.194377 22108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1726) took 2313 ms. Trace:
W20251023 02:05:33.951450 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 67435.6us
W20251023 02:05:34.349248 22110 rpcz_store.cc:269] 1023 02:05:31.683613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.750983 (+ 67370us) service_pool.cc:225] Handling call
1023 02:05:34.034502 (+2283519us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b54002fc1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:34.034625 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1595132672}
I20251023 02:05:34.351016 22110 raft_consensus.cc:1240] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 8. Current term is 9. Ops: []
W20251023 02:05:34.347363 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:34.351613 21925 leader_election.cc:304] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:34.351513 24853 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:34.352237 22110 raft_consensus.cc:1240] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 8. Current term is 9. Ops: []
W20251023 02:05:34.353725 24864 consensus_meta.cc:287] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.600s user 0.007s sys 0.003s
I20251023 02:05:34.353952 24853 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:34.030908 24894 log.cc:927] Time spent T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.230s user 0.000s sys 0.000s
I20251023 02:05:34.347822 24872 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:34.297886 24887 log.cc:927] Time spent T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.416s user 0.001s sys 0.000s
I20251023 02:05:34.355819 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:34.350661 24346 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.913512 24627 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.357067 22036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1333) took 2490 ms. Trace:
I20251023 02:05:34.354355 24864 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.879932 24871 leader_election.cc:290] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.365769 22036 rpcz_store.cc:276] 1023 02:05:31.866745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.059448 (+192703us) service_pool.cc:225] Handling call
1023 02:05:32.073935 (+ 14487us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:32.074263 (+ 328us) raft_consensus.cc:1534] Early marking committed up to index 3
1023 02:05:32.247374 (+173111us) log.cc:844] Serialized 14 byte log entry
1023 02:05:33.739536 (+1492162us) spinlock_profiling.cc:243] Waited 4.62 ms on lock 0x7b48001701d8. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c649a7971 00007f9c6499ea37 00007f9c6499e8ca 00007f9c64a95a0f 00007f9c64a956d7 00007f9c64aa9cfc 00007f9c64aa9ca8 00007f9c64aa9c28 00007f9c64aa9be3 00007f9c64aa8e8c 00007f9c639e5326
1023 02:05:33.777649 (+ 38113us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:33.778111 (+ 462us) log.cc:844] Serialized 26 byte log entry
1023 02:05:33.778262 (+ 151us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:33.778298 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:33.778363 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:34.356673 (+578310us) raft_consensus.cc:1685] finished
1023 02:05:34.356812 (+ 139us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.356993 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":120629,"prepare.queue_time_us":435854,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":59,"raft.queue_time_us":233969,"raft.run_cpu_time_us":4617,"raft.run_wall_time_us":1511182,"spinlock_wait_cycles":10624384,"thread_start_us":1413046,"threads_started":2,"wal-append.queue_time_us":1525564}
I20251023 02:05:34.349607 22108 rpcz_store.cc:276] 1023 02:05:31.880679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.156285 (+275606us) service_pool.cc:225] Handling call
1023 02:05:32.156651 (+ 366us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:32.162999 (+ 6348us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:32.163031 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:32.163478 (+ 447us) log.cc:844] Serialized 26 byte log entry
1023 02:05:32.268130 (+104652us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:32.268172 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:32.268245 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:34.194166 (+1925921us) raft_consensus.cc:1685] finished
1023 02:05:34.194187 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.194320 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5942,"raft.queue_time_us":6959,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":5017,"thread_start_us":103952,"threads_started":1,"wal-append.queue_time_us":1132080}
I20251023 02:05:34.345461 24897 raft_consensus.cc:493] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:34.347990 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 503919us
I20251023 02:05:34.382930 22002 leader_election.cc:304] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:34.384861 22002 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:34.041504 24891 log.cc:927] Time spent T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.321s user 0.002s sys 0.001s
I20251023 02:05:34.357920 24808 raft_consensus.cc:2749] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:33.890323 24861 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:34.031498 24896 log.cc:927] Time spent T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.145s user 0.000s sys 0.000s
W20251023 02:05:34.347082 24762 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.042s user 0.001s sys 0.045s
W20251023 02:05:34.030455 24895 log.cc:927] Time spent T 00301d3b4a804bf19bad66c51fe19faa P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
W20251023 02:05:33.914363 24792 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.002s sys 0.004s
I20251023 02:05:33.912184 24798 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.030205 24807 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.357888 24760 consensus_queue.cc:1059] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20251023 02:05:34.398396 24807 raft_consensus.cc:2749] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251023 02:05:34.398972 24807 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20251023 02:05:34.397764 24792 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.069s user 0.012s sys 0.017s
I20251023 02:05:34.399595 24757 consensus_queue.cc:1059] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20251023 02:05:34.362964 24853 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.340880 24689 raft_consensus.cc:493] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:33.902927 24368 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.029031 24655 raft_consensus.cc:2749] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:33.913625 24902 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251023 02:05:34.042232 24823 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:34.043341 24624 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.357527 24744 raft_consensus.cc:493] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.930758 24754 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.357537 24900 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.193305 24640 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.361159 24793 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251023 02:05:34.330477 24880 consensus_meta.cc:287] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.586s user 0.010s sys 0.000s
I20251023 02:05:34.406248 24853 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.424571 24880 raft_consensus.cc:2749] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251023 02:05:34.424831 24692 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.363910 24851 raft_consensus.cc:3060] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:34.425189 24744 raft_consensus.cc:515] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.405025 24790 consensus_queue.cc:260] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.425916 24853 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:33.958505 24885 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:34.426483 24900 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:34.357671 24904 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:33.928850 24876 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.394520 24898 raft_consensus.cc:2804] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:34.396385 24813 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.397801 24789 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:34.366442 24748 consensus_queue.cc:1175] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "818907d63f1f4d84ba267ebe6ee837e0" responder_term: 7 status { last_received { term: 2 index: 2 } last_committed_idx: 2 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20251023 02:05:34.427780 24904 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:34.361357 24892 raft_consensus.cc:697] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:34.430198 24900 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.430356 24275 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.411722 24668 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.430917 24892 consensus_queue.cc:237] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.431193 24668 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:33.925061 24576 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.359287 24677 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:34.392292 24714 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:34.386051 24897 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:34.362097 24670 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:34.413666 24292 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:34.484061 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 123604us
I20251023 02:05:34.356985 21928 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:34.517820 24346 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.386873 22002 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:34.556866 24917 raft_consensus.cc:2749] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20251023 02:05:34.557876 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 170989us
I20251023 02:05:34.559360 22002 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:34.488633 22034 raft_consensus.cc:1240] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:34.490516 22032 raft_consensus.cc:1240] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
W20251023 02:05:34.492316 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 124694us
I20251023 02:05:34.396785 22107 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1719) took 2868 ms. Trace:
I20251023 02:05:34.526590 24748 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.562379 24917 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:34.562964 24748 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.563529 22076 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:34.490538 22000 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.81.3:33899 timed out after 1.733s (SENT) [suppressed 12 similar messages]
I20251023 02:05:34.494899 22077 leader_election.cc:304] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:34.562657 22107 rpcz_store.cc:276] 1023 02:05:31.528467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.150680 (+622213us) service_pool.cc:225] Handling call
1023 02:05:32.151077 (+ 397us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:32.151491 (+ 414us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:32.151532 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:32.152090 (+ 558us) log.cc:844] Serialized 26 byte log entry
1023 02:05:33.068719 (+916629us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b1800455920. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9cdd37 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c5f62f2bb 00007f9c5f634dc3 00007f9c5f635171 00007f9c5fa5465f
1023 02:05:33.506139 (+437420us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:33.544143 (+ 38004us) log.cc:844] Serialized 14 byte log entry
1023 02:05:33.544341 (+ 198us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:33.545312 (+ 971us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:34.396514 (+851202us) raft_consensus.cc:1685] finished
1023 02:05:34.396608 (+ 94us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.396729 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":261241,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":1631,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":14181,"spinlock_wait_cycles":8671616,"thread_start_us":437216,"threads_started":1,"wal-append.queue_time_us":1653556}
W20251023 02:05:34.566366 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 76011.6us
W20251023 02:05:34.567231 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 72911.7us
I20251023 02:05:34.568264 24853 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.427232 24744 leader_election.cc:290] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.495141 22036 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "774c010489aa4775bb6d636926900529"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:34.386780 22037 raft_consensus.cc:2468] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:34.573793 22076 leader_election.cc:400] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7.
I20251023 02:05:34.574188 22076 leader_election.cc:403] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20251023 02:05:34.496507 24789 consensus_queue.cc:1059] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:05:34.575611 24789 raft_consensus.cc:3055] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:34.576085 24789 raft_consensus.cc:740] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:34.576465 22036 raft_consensus.cc:606] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:34.576930 22036 raft_consensus.cc:614] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:34.578136 24851 raft_consensus.cc:2749] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499) with higher term. Message: Invalid argument: T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7.
I20251023 02:05:34.498289 24677 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:34.387238 22589 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:34.579110 24748 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:34.406611 24689 raft_consensus.cc:3060] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:34.580583 22616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 393) took 1961 ms. Trace:
W20251023 02:05:34.581008 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 193741us
I20251023 02:05:34.378000 24694 consensus_queue.cc:1048] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20251023 02:05:34.583851 24677 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:34.435132 24714 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.398761 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:34.581143 22616 rpcz_store.cc:276] 1023 02:05:32.619285 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.622596 (+ 3311us) service_pool.cc:225] Handling call
1023 02:05:32.666318 (+ 43722us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:33.731812 (+1065494us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.579847 (+848035us) spinlock_profiling.cc:243] Waited 840 ms on lock 0x7b54005fdf54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:34.580278 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1931573120}
I20251023 02:05:34.585999 22617 raft_consensus.cc:2393] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 6: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:34.586642 24610 raft_consensus.cc:3055] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 LEADER]: Stepping down as leader of term 8
I20251023 02:05:34.587178 24610 raft_consensus.cc:740] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:34.587261 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:34.401469 22618 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:34.588733 24796 consensus_queue.cc:1059] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251023 02:05:34.590004 24796 consensus_queue.cc:1243] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer 1ab8b0b58147446d84d901b4ed2ff39e log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20251023 02:05:34.506763 22035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1324) took 2645 ms (client timeout 1660 ms). Trace:
I20251023 02:05:34.590366 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:34.590840 24641 raft_consensus.cc:3055] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Stepping down as leader of term 8
I20251023 02:05:34.591346 24641 raft_consensus.cc:740] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:34.577442 24789 consensus_queue.cc:260] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.587276 24714 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.365360 24543 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
W20251023 02:05:34.424687 24547 consensus_meta.cc:287] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.690s user 0.013s sys 0.000s
I20251023 02:05:34.430974 24578 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.592948 24789 raft_consensus.cc:3060] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:34.414135 24691 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.595788 24691 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.415553 24822 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:34.510438 24193 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.598753 24193 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:34.513048 24292 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:34.599318 24859 log.cc:927] Time spent T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 1.976s user 0.000s sys 0.009s
I20251023 02:05:34.483681 21859 catalog_manager.cc:5649] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 3 to 5, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 5 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:34.532876 24655 raft_consensus.cc:493] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:34.522979 24791 consensus_queue.cc:260] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.614741 24791 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:34.614529 24655 raft_consensus.cc:515] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } attrs { replace: true } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:34.536624 24662 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.520679 24368 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.539888 24813 raft_consensus.cc:2749] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:34.522063 24668 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.527506 22615 raft_consensus.cc:1240] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
W20251023 02:05:34.541754 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 184706us
I20251023 02:05:34.406107 24850 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:34.543056 22029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1321) took 2725 ms (client timeout 1771 ms). Trace:
I20251023 02:05:34.544204 22030 raft_consensus.cc:1240] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:34.623627 24850 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:34.624179 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 65123.9us
I20251023 02:05:34.624686 21928 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:33.922395 24879 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.626098 24744 consensus_queue.cc:1048] T 496e171acd9c4d6d9dab2ab594c06e57 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:05:34.627372 22002 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:34.627147 24514 consensus_queue.cc:260] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:34.623343 22029 rpcz_store.cc:269] 1023 02:05:31.817600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.926784 (+109184us) service_pool.cc:225] Handling call
1023 02:05:34.542523 (+2615739us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b540036d154. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:34.542825 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":578236288}
I20251023 02:05:34.628374 24879 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.628948 24514 raft_consensus.cc:3060] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:34.436091 24885 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.629523 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:34.432999 24898 raft_consensus.cc:697] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:34.630407 22029 raft_consensus.cc:2468] T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:34.631814 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:34.631215 21859 catalog_manager.cc:5649] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 4 to 6, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 6 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: HEALTHY } } }
I20251023 02:05:34.632674 22029 raft_consensus.cc:2393] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 9: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:34.631961 24898 consensus_queue.cc:237] T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.541306 22033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1561) took 1972 ms. Trace:
I20251023 02:05:34.431485 24904 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:34.414389 24905 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.451743 24897 raft_consensus.cc:515] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.285667 24868 raft_consensus.cc:493] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:34.560248 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:34.569413 24853 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:34.570984 24919 raft_consensus.cc:2749] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20251023 02:05:34.392686 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:34.672772 22111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1619) took 1039 ms. Trace:
I20251023 02:05:34.578574 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:34.674137 22036 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:34.673549 22111 rpcz_store.cc:276] 1023 02:05:33.632724 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:34.358248 (+725524us) service_pool.cc:225] Handling call
1023 02:05:34.358525 (+ 277us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:34.358740 (+ 215us) raft_consensus.cc:1534] Early marking committed up to index 7
1023 02:05:34.493873 (+135133us) log.cc:844] Serialized 14 byte log entry
1023 02:05:34.669663 (+175790us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:34.670621 (+ 958us) raft_consensus.cc:1645] Marking committed up to 7
1023 02:05:34.670942 (+ 321us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.671395 (+ 453us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.672087 (+ 692us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2808,"prepare.queue_time_us":42273,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":120,"raft.queue_time_us":6169,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":166,"thread_start_us":295086,"threads_started":2}
I20251023 02:05:34.674846 24792 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251023 02:05:34.675526 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:34.675946 24849 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:34.676112 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.676410 22036 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:34.676821 22111 raft_consensus.cc:2468] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 10.
I20251023 02:05:34.676723 24849 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.373634 24627 raft_consensus.cc:515] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:34.505779 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:34.508332 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:34.678645 22621 raft_consensus.cc:2410] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20251023 02:05:34.590574 22035 rpcz_store.cc:269] 1023 02:05:31.860865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.906614 (+ 45749us) service_pool.cc:225] Handling call
1023 02:05:34.506239 (+2599625us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540026eb54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:34.506469 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":474362240}
I20251023 02:05:34.679695 22035 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:34.679890 24627 leader_election.cc:290] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.680374 22035 raft_consensus.cc:606] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Received request to transfer leadership to TS ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:34.681008 22035 raft_consensus.cc:614] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:34.604082 24691 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:34.602208 24193 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.633754 22033 rpcz_store.cc:276] 1023 02:05:32.568691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:33.063574 (+494883us) service_pool.cc:225] Handling call
1023 02:05:33.064513 (+ 939us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:33.615738 (+551225us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:33.630244 (+ 14506us) log.cc:844] Serialized 14 byte log entry
1023 02:05:33.772077 (+141833us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:33.772126 (+ 49us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:33.772170 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.540732 (+768562us) spinlock_profiling.cc:243] Waited 442 ms on lock 0x7b540001e0d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:34.540803 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.541074 (+ 271us) inbound_call.cc:173] Queueing success response
1023 02:05:34.612061 (+ 70987us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
Metrics: {"mutex_wait_us":43657,"prepare.queue_time_us":19424,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":28626,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":12341,"spinlock_wait_cycles":1496313344,"thread_start_us":119347,"threads_started":1,"wal-append.queue_time_us":256016}
I20251023 02:05:34.685165 22035 raft_consensus.cc:1275] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251023 02:05:34.685961 22035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 308) took 1799 ms. Trace:
I20251023 02:05:34.686357 22035 rpcz_store.cc:276] 1023 02:05:32.886987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:34.683221 (+1796234us) service_pool.cc:225] Handling call
1023 02:05:34.684754 (+ 1533us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:34.685641 (+ 887us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.685880 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:34.687927 22033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 309) took 1800 ms. Trace:
I20251023 02:05:34.689200 21960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53744 (request call id 1332) took 1129 ms. Trace:
W20251023 02:05:34.689930 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 65605us
I20251023 02:05:34.690389 24905 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:34.599798 24870 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.691886 24870 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.605962 24577 leader_election.cc:290] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:34.582525 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:34.606845 24641 consensus_queue.cc:260] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:34.608520 24899 log.cc:927] Time spent T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.696s user 0.004s sys 0.017s
I20251023 02:05:34.588799 24610 consensus_queue.cc:260] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:34.610446 24907 log.cc:927] Time spent T 3c29a39800fe46e9b0b85ea73ecc703f P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.376s user 0.006s sys 0.005s
I20251023 02:05:34.692435 21928 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:34.611363 24677 consensus_queue.cc:1048] T 972d07041fbd4097854ef23c73f09c75 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20251023 02:05:34.613011 24903 log.cc:927] Time spent T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.655s user 0.000s sys 0.009s
I20251023 02:05:34.612898 22109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1716) took 3249 ms. Trace:
I20251023 02:05:34.613416 24292 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.610488 24920 raft_consensus.cc:2804] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election won for term 9
I20251023 02:05:34.417737 24864 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:34.745214 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 53082us
I20251023 02:05:34.746335 21928 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:34.747339 24368 raft_consensus.cc:2749] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251023 02:05:34.633541 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:34.701503 24853 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:34.704836 21962 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53744 (request call id 1331) took 1145 ms. Trace:
I20251023 02:05:34.750247 22029 raft_consensus.cc:2368] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 9. Current term is 10.
I20251023 02:05:34.750306 21962 rpcz_store.cc:276] 1023 02:05:33.558792 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:33.558978 (+ 186us) service_pool.cc:225] Handling call
1023 02:05:33.580104 (+ 21126us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:33.626039 (+ 45935us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.704190 (+1078151us) spinlock_profiling.cc:243] Waited 40.7 ms on lock 0x7b54000bf4d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:34.704555 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93708544}
I20251023 02:05:34.751037 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:34.751291 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:34.752106 22029 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:34.752532 24507 leader_election.cc:290] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:34.646647 22589 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:34.753122 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:34.701344 24641 raft_consensus.cc:3060] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:34.652908 24744 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:34.538086 24667 raft_consensus.cc:3055] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:34.802102 24667 raft_consensus.cc:740] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:34.803310 24667 consensus_queue.cc:260] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.804728 24667 raft_consensus.cc:3060] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:34.604899 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:34.601989 24862 log.cc:927] Time spent T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 1.875s user 0.000s sys 0.003s
I20251023 02:05:34.714026 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:34.807904 22620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34820 (request call id 401) took 2872 ms. Trace:
W20251023 02:05:34.809852 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:34.714834 24923 log.cc:927] Time spent T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.260s user 0.000s sys 0.001s
I20251023 02:05:34.716202 24897 leader_election.cc:290] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.689711 21960 rpcz_store.cc:276] 1023 02:05:33.559468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:33.559583 (+ 115us) service_pool.cc:225] Handling call
1023 02:05:34.688719 (+1129136us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b540030e8d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:34.688941 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":648937344}
I20251023 02:05:34.665892 24813 raft_consensus.cc:2749] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251023 02:05:34.716699 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:34.696645 21959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1684) took 2685 ms (client timeout 2005 ms). Trace:
I20251023 02:05:34.717489 24872 raft_consensus.cc:3055] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Stepping down as leader of term 7
W20251023 02:05:34.719027 24906 log.cc:927] Time spent T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.485s user 0.000s sys 0.010s
W20251023 02:05:34.698199 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 71010.4us
I20251023 02:05:34.712896 24689 raft_consensus.cc:515] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.698926 22590 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:34.695466 24870 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:34.704107 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1604) took 2084 ms. Trace:
I20251023 02:05:34.698276 22035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 310) took 1810 ms. Trace:
I20251023 02:05:34.700513 22037 raft_consensus.cc:1240] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 4. Current term is 5. Ops: []
I20251023 02:05:34.688496 22033 rpcz_store.cc:276] 1023 02:05:32.887308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:34.685852 (+1798544us) service_pool.cc:225] Handling call
1023 02:05:34.686425 (+ 573us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:34.686904 (+ 479us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:34.686947 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:34.687034 (+ 87us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:34.687609 (+ 575us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.687698 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.687858 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:34.701835 24850 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:34.703908 24610 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:34.744396 24627 raft_consensus.cc:2749] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:34.744531 24920 raft_consensus.cc:697] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:34.708335 21961 raft_consensus.cc:1217] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-7.5] Dedup: 5.4->[7.5-7.5]
I20251023 02:05:34.618171 24655 leader_election.cc:290] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:34.646955 22104 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39662 (request call id 1727) took 2669 ms (client timeout 1541 ms). Trace:
I20251023 02:05:34.739645 22109 rpcz_store.cc:276] 1023 02:05:31.363496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.142259 (+778763us) service_pool.cc:225] Handling call
1023 02:05:32.194554 (+ 52295us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:32.195379 (+ 825us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.612381 (+2417002us) spinlock_profiling.cc:243] Waited 51.9 ms on lock 0x7b54002d05d0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:34.612633 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":119280512}
I20251023 02:05:34.758074 22029 raft_consensus.cc:2410] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20251023 02:05:34.754230 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1543) took 2541 ms. Trace:
I20251023 02:05:34.829646 24507 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.633212 21859 catalog_manager.cc:5649] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 4 to 6, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 6 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } }
I20251023 02:05:34.762778 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:34.763365 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:34.763785 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:34.764453 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:34.765488 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:34.703171 24868 raft_consensus.cc:515] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:34.851347 22587 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:34.853123 24868 leader_election.cc:290] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.783958 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.854061 22111 raft_consensus.cc:2468] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:34.855304 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.856170 22111 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:34.785008 24905 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:34.706782 24921 log.cc:927] Time spent T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.252s user 0.002s sys 0.000s
I20251023 02:05:34.859022 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:34.708369 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1714) took 3355 ms. Trace:
I20251023 02:05:34.860358 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:34.861752 21956 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:34.861274 24920 consensus_queue.cc:237] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.860210 22106 rpcz_store.cc:276] 1023 02:05:31.352522 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.098031 (+745509us) service_pool.cc:225] Handling call
1023 02:05:32.120848 (+ 22817us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:32.121375 (+ 527us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:32.121420 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:32.129243 (+ 7823us) log.cc:844] Serialized 26 byte log entry
1023 02:05:32.252602 (+123359us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:32.494234 (+241632us) log.cc:844] Serialized 14 byte log entry
1023 02:05:33.646198 (+1151964us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:33.646300 (+ 102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:34.598186 (+951886us) spinlock_profiling.cc:243] Waited 3.68 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:34.707767 (+109581us) raft_consensus.cc:1685] finished
1023 02:05:34.707910 (+ 143us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.708146 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":380368,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":37,"raft.queue_time_us":112067,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":28550,"spinlock_wait_cycles":8465664,"thread_start_us":1368358,"threads_started":3,"wal-append.queue_time_us":386602}
I20251023 02:05:34.862924 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.863674 22106 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:34.864751 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.865762 22102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1618) took 1238 ms. Trace:
I20251023 02:05:34.865965 24740 consensus_queue.cc:1059] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:05:34.867241 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1615) took 1241 ms. Trace:
I20251023 02:05:34.866175 22102 rpcz_store.cc:276] 1023 02:05:33.626970 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:34.360148 (+733178us) service_pool.cc:225] Handling call
1023 02:05:34.360406 (+ 258us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:34.360603 (+ 197us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:34.681374 (+320771us) log.cc:844] Serialized 14 byte log entry
1023 02:05:34.829191 (+147817us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:34.840667 (+ 11476us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:34.849173 (+ 8506us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.865441 (+ 16268us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.865590 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2154,"prepare.queue_time_us":60078,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":741,"raft.queue_time_us":2041,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":19574,"thread_start_us":441713,"threads_started":2}
I20251023 02:05:34.868319 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.749341 21928 leader_election.cc:304] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:34.707139 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 44833.8us
W20251023 02:05:34.754135 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 142693us
I20251023 02:05:34.827106 22036 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 8.
I20251023 02:05:34.922708 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:34.706465 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:34.924602 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:34.925158 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:34.925418 21953 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:34.926344 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:34.707844 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1556) took 2193 ms. Trace:
I20251023 02:05:34.927547 22587 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:34.927671 22030 rpcz_store.cc:276] 1023 02:05:32.514448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.797919 (+283471us) service_pool.cc:225] Handling call
1023 02:05:34.543644 (+1745725us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:34.623875 (+ 80231us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.707622 (+ 83747us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b540036d150. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9844d 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:34.707769 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4014304640}
W20251023 02:05:34.738735 24831 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.438s user 0.000s sys 0.026s
I20251023 02:05:34.840324 22035 rpcz_store.cc:276] 1023 02:05:32.887595 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:34.687190 (+1799595us) service_pool.cc:225] Handling call
1023 02:05:34.696985 (+ 9795us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:34.697897 (+ 912us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:34.697921 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:34.697985 (+ 64us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:34.698016 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.698084 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.698209 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:34.844367 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 145694us
W20251023 02:05:34.839485 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 187338us
I20251023 02:05:34.839812 22103 rpcz_store.cc:276] 1023 02:05:32.619489 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.734252 (+114763us) service_pool.cc:225] Handling call
1023 02:05:32.734514 (+ 262us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:33.752627 (+1018113us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.703575 (+950948us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b540032fad4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:34.703923 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2035541760}
I20251023 02:05:34.846522 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1566) took 1954 ms. Trace:
I20251023 02:05:34.847410 21961 raft_consensus.cc:1240] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: [3.3-7.5]
I20251023 02:05:34.829329 24872 raft_consensus.cc:740] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:34.760535 24864 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251023 02:05:34.857710 24543 consensus_queue.cc:1059] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20251023 02:05:34.867331 24740 raft_consensus.cc:3055] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Stepping down as leader of term 8
I20251023 02:05:34.936422 24924 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.936683 24740 raft_consensus.cc:740] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:34.789287 24791 consensus_queue.cc:1048] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20251023 02:05:34.848588 22104 rpcz_store.cc:269] 1023 02:05:31.977769 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.985451 (+ 7682us) service_pool.cc:225] Handling call
1023 02:05:34.618973 (+2633522us) spinlock_profiling.cc:243] Waited 32.7 ms on lock 0x7b54001abdd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:34.646646 (+ 27673us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":75141504}
I20251023 02:05:34.870218 21958 raft_consensus.cc:2468] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:34.868348 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:34.849248 22028 rpcz_store.cc:276] 1023 02:05:32.212718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.620584 (+407866us) service_pool.cc:225] Handling call
1023 02:05:32.620918 (+ 334us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:32.895964 (+275046us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:32.896016 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:32.896736 (+ 720us) log.cc:844] Serialized 26 byte log entry
1023 02:05:34.606340 (+1709604us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b180043d0e0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5c9cdd37 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c5f62f2bb 00007f9c5f634dc3 00007f9c5f635171 00007f9c5fa5465f
1023 02:05:34.675614 (+ 69274us) raft_consensus.cc:1645] Marking committed up to 2
1023 02:05:34.675687 (+ 73us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.677875 (+ 2188us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:34.753644 (+ 75769us) raft_consensus.cc:1685] finished
1023 02:05:34.753670 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.754024 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14077,"raft.queue_time_us":315272,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":8251,"spinlock_wait_cycles":29272320,"thread_start_us":320733,"threads_started":2,"wal-append.queue_time_us":1731817}
I20251023 02:05:34.868954 21859 catalog_manager.cc:5649] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:34.871703 21962 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:34.861297 22029 raft_consensus.cc:2468] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:34.868469 21955 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:34.857457 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:34.874265 21925 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:34.867679 22105 rpcz_store.cc:276] 1023 02:05:33.626174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:33.933372 (+307198us) service_pool.cc:225] Handling call
1023 02:05:34.031654 (+ 98282us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:34.032111 (+ 457us) raft_consensus.cc:1534] Early marking committed up to index 4
1023 02:05:34.358768 (+326657us) log.cc:844] Serialized 14 byte log entry
1023 02:05:34.496811 (+138043us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:34.497893 (+ 1082us) log.cc:844] Serialized 26 byte log entry
1023 02:05:34.498136 (+ 243us) raft_consensus.cc:1645] Marking committed up to 4
1023 02:05:34.498201 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.498347 (+ 146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:34.705537 (+207190us) spinlock_profiling.cc:243] Waited 97.8 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:34.866716 (+161179us) raft_consensus.cc:1685] finished
1023 02:05:34.866886 (+ 170us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.867058 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":326428,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":46935,"raft.run_cpu_time_us":427,"raft.run_wall_time_us":576,"spinlock_wait_cycles":224929664,"thread_start_us":137335,"threads_started":1,"wal-append.queue_time_us":95009}
I20251023 02:05:34.937202 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:34.936700 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.882426 21956 raft_consensus.cc:2468] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:34.808254 22620 rpcz_store.cc:276] 1023 02:05:31.934973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:31.935149 (+ 176us) service_pool.cc:225] Handling call
1023 02:05:31.942719 (+ 7570us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:32.143755 (+201036us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:32.149055 (+ 5300us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:32.225355 (+ 76300us) log.cc:844] Serialized 26 byte log entry
1023 02:05:32.406975 (+181620us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:32.460747 (+ 53772us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:33.074793 (+614046us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b540052d854. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:33.074863 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:34.600924 (+1526061us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:34.807530 (+206606us) raft_consensus.cc:1685] finished
1023 02:05:34.807549 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.807725 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17495,"raft.queue_time_us":36136,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":11851,"spinlock_wait_cycles":396667136,"thread_start_us":131092,"threads_started":1,"wal-append.queue_time_us":400434}
I20251023 02:05:34.941352 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:34.942332 21962 raft_consensus.cc:2410] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251023 02:05:34.884371 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:34.939604 22103 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:34.944043 22109 raft_consensus.cc:2393] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 9: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:34.857724 24677 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:34.889559 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:34.866019 24576 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:34.947198 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:34.943316 22621 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:34.891552 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.891844 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:34.890833 21957 raft_consensus.cc:2468] T 496e171acd9c4d6d9dab2ab594c06e57 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:34.851358 24507 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:34.897305 24655 raft_consensus.cc:493] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.902667 24744 raft_consensus.cc:2749] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:34.908098 24920 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.907449 24689 leader_election.cc:290] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.913848 24193 raft_consensus.cc:493] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.914356 22110 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:34.919215 24879 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:34.921206 22589 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:34.828382 21959 rpcz_store.cc:269] 1023 02:05:32.011309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.039709 (+ 28400us) service_pool.cc:225] Handling call
1023 02:05:34.696081 (+2656372us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b540017dc54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:34.696391 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3668716416}
I20251023 02:05:34.905992 24323 raft_consensus.cc:3055] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Stepping down as leader of term 3
I20251023 02:05:34.945168 24792 raft_consensus.cc:2804] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251023 02:05:34.931723 22037 rpcz_store.cc:276] 1023 02:05:32.887879 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:34.699614 (+1811735us) service_pool.cc:225] Handling call
1023 02:05:34.700180 (+ 566us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:34.842413 (+142233us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.842653 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:34.931176 24831 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.722s user 0.000s sys 0.027s
I20251023 02:05:34.872515 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:34.937106 24924 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.944216 22102 raft_consensus.cc:2376] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7. Re-sending same reply.
I20251023 02:05:34.945643 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:34.958292 21961 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:34.959592 22107 raft_consensus.cc:2376] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 10. Re-sending same reply.
I20251023 02:05:34.943080 24558 raft_consensus.cc:3060] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:34.964437 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 90403.4us
I20251023 02:05:34.964579 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.965062 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:34.958811 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.965747 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.977248 22111 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:34.966320 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.929052 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:34.978900 22030 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:34.968395 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:34.947283 24677 raft_consensus.cc:493] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:34.949807 21859 catalog_manager.cc:5649] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 5 to 6, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 6 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:34.970145 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.981953 21955 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 8. Current term is 9. Ops: []
I20251023 02:05:34.981861 22029 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:35.028240 22029 raft_consensus.cc:606] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:35.028671 22029 raft_consensus.cc:614] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251023 02:05:34.983860 22035 raft_consensus.cc:1275] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20251023 02:05:34.984018 22033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 311) took 1276 ms. Trace:
I20251023 02:05:34.984858 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:35.030586 22033 rpcz_store.cc:276] 1023 02:05:33.707266 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:34.982482 (+1275216us) service_pool.cc:225] Handling call
1023 02:05:34.982950 (+ 468us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:34.983504 (+ 554us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:34.983543 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1023 02:05:34.983631 (+ 88us) raft_consensus.cc:1645] Marking committed up to 8
1023 02:05:34.983691 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.983787 (+ 96us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:34.983930 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2405,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":80}
I20251023 02:05:34.984644 24879 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.985541 21960 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:34.986212 24798 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:34.987465 22036 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:34.988298 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.989802 21958 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 6.
I20251023 02:05:34.969048 24323 raft_consensus.cc:740] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:34.990290 24924 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:34.990504 24655 raft_consensus.cc:515] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.990893 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:34.991931 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:35.047019 22111 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:34.993381 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:34.993824 24876 raft_consensus.cc:2804] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:34.995909 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:34.996485 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 67515.7us
I20251023 02:05:34.974084 24831 raft_consensus.cc:697] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:34.971812 24898 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:34.997012 24292 consensus_queue.cc:1059] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251023 02:05:34.997524 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 248399us
I20251023 02:05:34.973737 24193 raft_consensus.cc:515] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:34.998368 24897 consensus_queue.cc:1059] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:34.999354 22037 raft_consensus.cc:1240] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 7. Current term is 8. Ops: []
W20251023 02:05:35.000188 22077 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:35.000270 24933 log.cc:927] Time spent T 74071b399ecc41808f0ac84dc836af4c P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20251023 02:05:35.001080 22000 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.003654 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1559) took 2464 ms. Trace:
W20251023 02:05:35.004880 24916 log.cc:927] Time spent T 90dcf9b1f8874437a7cd406b2ace7eeb P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.566s user 0.002s sys 0.002s
I20251023 02:05:35.006395 24849 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:35.007411 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.008771 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.009979 21953 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251023 02:05:35.011791 21925 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.012799 24872 consensus_queue.cc:260] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.015774 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:35.016750 24883 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:35.017810 24576 raft_consensus.cc:515] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.021296 24909 raft_consensus.cc:2804] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:35.025430 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:34.969440 24792 raft_consensus.cc:493] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.041182 22029 raft_consensus.cc:1275] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 2. (index mismatch)
I20251023 02:05:35.038659 22103 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:34.989507 24740 consensus_queue.cc:260] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.029781 24677 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:35.129734 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:35.130347 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.132473 24292 consensus_queue.cc:1243] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer 02c53ba06cc3445a8534a84aa76b1c14 log is divergent from this leader: its last log entry 6.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251023 02:05:35.030133 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.126757 21961 raft_consensus.cc:2410] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20251023 02:05:35.043393 22035 raft_consensus.cc:1217] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 4.3->[5.4-8.5] Dedup: 5.4->[8.5-8.5]
W20251023 02:05:35.044617 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.051079 22102 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:35.051630 22109 raft_consensus.cc:2468] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 6.
I20251023 02:05:35.051988 22033 raft_consensus.cc:1240] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:35.052417 22076 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.056334 22028 raft_consensus.cc:2410] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251023 02:05:35.067550 22104 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.083441 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1569) took 1377 ms. Trace:
I20251023 02:05:35.136744 22037 rpcz_store.cc:276] 1023 02:05:33.706285 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:34.998698 (+1292413us) service_pool.cc:225] Handling call
1023 02:05:34.999090 (+ 392us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:35.083113 (+ 84023us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:35.083357 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 02:05:35.084645 22077 leader_election.cc:304] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:35.117110 22106 raft_consensus.cc:2393] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 8: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:35.118372 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:35.080489 22032 rpcz_store.cc:276] 1023 02:05:32.538882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.983886 (+445004us) service_pool.cc:225] Handling call
1023 02:05:32.984287 (+ 401us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:34.562494 (+1578207us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:35.003311 (+440817us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540001ead4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:35.003588 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3462654336}
I20251023 02:05:35.063421 24898 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:35.122900 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 121972us
I20251023 02:05:35.053864 24831 consensus_queue.cc:237] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:35.126540 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 114845us
W20251023 02:05:35.127141 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.140290 22035 raft_consensus.cc:1240] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 8. Current term is 9. Ops: [5.4-8.5]
I20251023 02:05:35.124539 24883 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.127753 21928 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.128140 24813 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:35.146742 22108 raft_consensus.cc:2393] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 9: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:35.068277 24876 raft_consensus.cc:697] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:35.147107 21925 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:35.135147 24792 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 10 FOLLOWER]: Advancing to term 11
W20251023 02:05:35.147394 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 95084.5us
I20251023 02:05:35.148038 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:35.148578 22029 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:35.163426 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:35.159910 22037 raft_consensus.cc:1240] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 8. Current term is 9. Ops: []
W20251023 02:05:35.167042 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 245498us
W20251023 02:05:35.167963 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.163039 21959 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:35.143177 24898 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.165448 22105 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.126739 21859 catalog_manager.cc:5649] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:35.171531 22029 raft_consensus.cc:606] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:35.148197 24798 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:35.175707 22109 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20251023 02:05:35.142120 24670 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:35.185940 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.116683 24920 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.180282 22029 raft_consensus.cc:614] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251023 02:05:35.187120 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.159336 24292 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251023 02:05:35.190002 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.162786 24740 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
W20251023 02:05:35.199625 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 272259us
I20251023 02:05:35.197603 21928 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:35.207351 22000 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:35.207911 21926 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.169648 24813 raft_consensus.cc:697] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:35.175058 24789 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:35.220131 22000 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.222045 22109 raft_consensus.cc:1275] T 66c8e1c6015d47879ab7927548ad3b02 P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20251023 02:05:35.196507 24292 raft_consensus.cc:493] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:35.223088 21925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 82425.4us
W20251023 02:05:35.224539 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 225329us
I20251023 02:05:35.175541 24864 consensus_queue.cc:1048] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20251023 02:05:35.225395 22076 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:35.230098 21927 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.232461 24892 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:35.233256 24920 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:35.233018 24892 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:35.233819 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:35.234241 22074 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.235167 24892 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:35.235405 24853 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Leader election won for term 10
I20251023 02:05:35.235968 22103 raft_consensus.cc:1275] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251023 02:05:35.237999 24879 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:35.238127 24507 leader_election.cc:290] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:35.239189 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 154673us
I20251023 02:05:35.218416 24627 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:35.243726 22105 raft_consensus.cc:2468] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:35.216674 24789 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.253005 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:35.227258 24292 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
W20251023 02:05:35.261999 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.263782 21925 leader_election.cc:304] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:35.245344 22002 connection.cc:232] RPC call timeout handler was delayed by -1.4848428310000372s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2005s for a response
I20251023 02:05:35.222498 24670 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:35.265717 21959 raft_consensus.cc:2393] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 10: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:35.266170 24813 consensus_queue.cc:237] T c954cadc886d47b39ed2cf51f9ac59f7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.269942 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:35.239715 24641 raft_consensus.cc:2749] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:35.278373 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.279134 21961 raft_consensus.cc:2393] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 8: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:05:35.280165 24913 log.cc:927] Time spent T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.842s user 0.007s sys 0.000s
I20251023 02:05:35.281661 22076 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.239869 24558 raft_consensus.cc:2804] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
W20251023 02:05:35.283989 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 114814us
I20251023 02:05:35.258894 24346 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:35.276823 22077 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:35.271664 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 350656us
I20251023 02:05:35.275264 24689 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251023 02:05:35.273106 22109 raft_consensus.cc:1240] T e8a971ac884645548d12e7c83cc039bc P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251023 02:05:35.282320 22000 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.254556 24905 raft_consensus.cc:2804] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:35.254938 24789 raft_consensus.cc:3060] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.272782 24853 raft_consensus.cc:697] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:35.295229 24905 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.299037 24905 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.303979 22624 consensus_queue.cc:237] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.303575 24898 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.306421 22000 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:35.307689 22000 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:35.308565 24858 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:35.308995 24858 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:35.309350 24858 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.309355 24898 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:35.310730 24882 consensus_queue.cc:1048] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20251023 02:05:35.316648 24883 leader_election.cc:290] T 816ef481cee84b259bc409b45a81baf1 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:35.293112 24689 raft_consensus.cc:493] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.318142 24689 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:35.318531 24514 raft_consensus.cc:3055] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:35.318930 24514 raft_consensus.cc:740] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:35.319626 24858 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.319833 24514 consensus_queue.cc:260] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.320924 24514 raft_consensus.cc:3060] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:35.284056 24558 raft_consensus.cc:493] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.323684 22104 raft_consensus.cc:2468] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 8.
I20251023 02:05:35.274143 22105 raft_consensus.cc:1240] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:35.272768 24610 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:35.291164 22107 raft_consensus.cc:2393] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 9: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:35.291962 22587 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.327271 22032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:55760 (request call id 7) took 1569 ms. Trace:
I20251023 02:05:35.327785 22032 rpcz_store.cc:276] 1023 02:05:33.729051 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:35.176659 (+1447608us) service_pool.cc:225] Handling call
1023 02:05:35.298394 (+121735us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 02:05:35.330140 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 74901.3us
I20251023 02:05:35.309758 24744 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251023 02:05:35.337958 24744 raft_consensus.cc:493] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.338387 24744 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:35.293295 24868 raft_consensus.cc:2804] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:35.342067 24868 raft_consensus.cc:697] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:35.342702 24798 raft_consensus.cc:493] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.295415 21859 catalog_manager.cc:5649] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 2 to 4, leader changed from 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) to ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3). New cstate: current_term: 4 leader_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:35.280207 24876 consensus_queue.cc:237] T c0594e36e0824e13965498e412eee3fb P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.344422 22109 raft_consensus.cc:1240] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 LEADER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 8. Current term is 9. Ops: []
I20251023 02:05:35.334157 22111 raft_consensus.cc:2468] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:35.310271 24848 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:35.306747 24905 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.334760 24870 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:35.334844 22623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34806 (request call id 397) took 1555 ms. Trace:
I20251023 02:05:35.230386 24323 consensus_queue.cc:260] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.335232 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.335246 22076 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.326706 24850 raft_consensus.cc:2749] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251023 02:05:35.336552 22104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:35.343158 24798 raft_consensus.cc:3060] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
W20251023 02:05:35.347904 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 87810.5us
I20251023 02:05:35.296571 24655 leader_election.cc:290] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:35.352157 22105 raft_consensus.cc:1240] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 3. Current term is 4. Ops: []
I20251023 02:05:35.330166 24558 raft_consensus.cc:3060] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:35.355787 21928 leader_election.cc:304] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:35.357856 24920 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:35.358632 22104 raft_consensus.cc:2393] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:35.361569 22107 raft_consensus.cc:1275] T f745a2e5ae9f4a829fa01ad2bce79767 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251023 02:05:35.362622 21859 catalog_manager.cc:5649] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 3 to 4, leader changed from 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 4 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251023 02:05:35.347558 24848 raft_consensus.cc:493] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.332859 24610 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:35.366201 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.350937 24870 raft_consensus.cc:493] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.352581 24858 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:35.351152 22623 rpcz_store.cc:276] 1023 02:05:33.779770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:33.779903 (+ 133us) service_pool.cc:225] Handling call
1023 02:05:33.805393 (+ 25490us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:33.898977 (+ 93584us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:33.899008 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:33.899452 (+ 444us) log.cc:844] Serialized 26 byte log entry
1023 02:05:34.388964 (+489512us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:34.389008 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:34.698489 (+309481us) spinlock_profiling.cc:243] Waited 75.6 ms on lock 0x7b54005ba254. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:34.698575 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:35.279227 (+580652us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:35.331277 (+ 52050us) raft_consensus.cc:1685] finished
1023 02:05:35.331313 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:35.334690 (+ 3377us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17529,"raft.queue_time_us":44326,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":91581,"spinlock_wait_cycles":427139840,"thread_start_us":431521,"threads_started":1,"wal-append.queue_time_us":490114}
I20251023 02:05:35.372532 21961 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.373018 24796 consensus_queue.cc:1059] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251023 02:05:35.376407 21925 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.332731 24346 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.382062 24346 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:35.384573 24689 raft_consensus.cc:515] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.386035 24948 consensus_queue.cc:1048] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251023 02:05:35.343073 24868 consensus_queue.cc:237] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.387935 22076 leader_election.cc:304] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.361140 24920 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:35.370779 24848 raft_consensus.cc:3060] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:35.390235 21859 catalog_manager.cc:5649] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 1 to 5, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2). New cstate: current_term: 5 leader_uuid: "818907d63f1f4d84ba267ebe6ee837e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:35.392393 22111 raft_consensus.cc:1240] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251023 02:05:35.393898 24864 raft_consensus.cc:2804] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:35.402101 24864 raft_consensus.cc:493] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:35.401887 24853 consensus_queue.cc:237] T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.402523 24864 raft_consensus.cc:3060] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:35.405355 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.405680 22077 leader_election.cc:400] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Vote denied by peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645) with higher term. Message: Invalid argument: T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 9. Current term is 10.
I20251023 02:05:35.379155 24870 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:35.406963 24641 raft_consensus.cc:2804] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:35.407898 24791 consensus_queue.cc:1059] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251023 02:05:35.409690 24897 raft_consensus.cc:2749] T e32bb6c0622f44f8806f77eee0691856 P 02c53ba06cc3445a8534a84aa76b1c14 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251023 02:05:35.409754 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 68855.3us
I20251023 02:05:35.411425 21928 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:35.397161 24920 raft_consensus.cc:3060] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.414235 24627 raft_consensus.cc:2749] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:35.415122 24627 consensus_queue.cc:1048] T bd7f335f5df946c481e05b1a2a95f182 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251023 02:05:35.421864 22076 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.425985 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:35.427851 24796 consensus_queue.cc:1048] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251023 02:05:35.433915 24677 raft_consensus.cc:515] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.436669 24677 leader_election.cc:290] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:35.439831 22618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34806 (request call id 396) took 2549 ms (client timeout 1696 ms). Trace:
W20251023 02:05:35.440312 22618 rpcz_store.cc:269] 1023 02:05:32.890331 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.893050 (+ 2719us) service_pool.cc:225] Handling call
1023 02:05:35.439465 (+2546415us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54001531d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:35.439618 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3265911680}
I20251023 02:05:35.444283 24872 raft_consensus.cc:3060] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.450577 24924 raft_consensus.cc:3055] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:35.451010 24924 raft_consensus.cc:740] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:35.451884 24924 consensus_queue.cc:260] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 7.2, Last appended by leader: 2, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.452423 21859 catalog_manager.cc:5649] T de76561746ce42549f51431c3f40bed7 P 02c53ba06cc3445a8534a84aa76b1c14 reported cstate change: term changed from 5 to 6, leader changed from ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3) to 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1). New cstate: current_term: 6 leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20251023 02:05:35.454277 22034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38998 (request call id 1549) took 3084 ms. Trace:
I20251023 02:05:35.453141 24924 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.454834 24610 leader_election.cc:290] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:35.456028 24905 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:35.457873 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.458050 24689 leader_election.cc:290] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:35.458434 21953 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 5.
I20251023 02:05:35.461261 21953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.459728 24847 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:35.461871 22109 raft_consensus.cc:1240] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251023 02:05:35.459872 21958 raft_consensus.cc:1275] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251023 02:05:35.466181 22001 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.454660 22034 rpcz_store.cc:276] 1023 02:05:32.369804 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:32.765250 (+395446us) service_pool.cc:225] Handling call
1023 02:05:32.765615 (+ 365us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:34.566065 (+1800450us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:35.454028 (+887963us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54003a8c54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:35.454220 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3961273216}
I20251023 02:05:35.416909 24627 raft_consensus.cc:2749] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:35.478562 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 85720.5us
I20251023 02:05:35.484819 22111 raft_consensus.cc:1240] T 770eed9a6198463c889a1b9a978bd182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:35.487396 22076 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:35.488751 22076 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:35.500726 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 87669us
I20251023 02:05:35.490041 24576 leader_election.cc:290] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:35.534034 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 259503us
I20251023 02:05:35.482900 24878 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251023 02:05:35.484647 24883 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:35.541934 24883 raft_consensus.cc:493] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.542341 24883 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:35.541442 24878 raft_consensus.cc:493] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.545102 21926 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:35.547385 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:35.549898 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 240447us
I20251023 02:05:35.536078 24507 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:35.547049 24905 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:35.567076 21958 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:35.530421 24796 consensus_queue.cc:1059] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20251023 02:05:35.557473 24878 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Advancing to term 10
W20251023 02:05:35.608153 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 63209.9us
I20251023 02:05:35.602145 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.597550 22109 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9. Ops: []
I20251023 02:05:35.590386 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.596258 22030 raft_consensus.cc:1240] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: [6.2-6.2]
I20251023 02:05:35.590936 21960 raft_consensus.cc:1240] T 8058bbefb0b74ae5949474c893a6947a P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20251023 02:05:35.595834 22077 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:35.566802 24507 raft_consensus.cc:493] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
W20251023 02:05:35.601866 21908 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 818907d63f1f4d84ba267ebe6ee837e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } }
I20251023 02:05:35.565729 24641 raft_consensus.cc:697] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Leader. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:35.578107 24905 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:35.655762 21958 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
W20251023 02:05:35.657541 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 374442us
I20251023 02:05:35.577622 24892 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:35.596817 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:35.590449 24908 log.cc:927] Time spent T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 1.239s user 0.003s sys 0.008s
I20251023 02:05:35.714840 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:35.715022 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 58617.2us
I20251023 02:05:35.715767 22111 raft_consensus.cc:2393] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 7: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:35.711203 21926 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:35.598371 24193 leader_election.cc:290] T 12db617f200e4104b5346432288ad306 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:35.592712 24909 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:35.698776 24507 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:35.740979 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1376a6af91b24e46859bf630618fcec3" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.589643 24689 consensus_queue.cc:1059] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.126s
I20251023 02:05:35.594229 24885 consensus_queue.cc:1059] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.144s
I20251023 02:05:35.761719 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:35.762354 22028 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:35.762501 24740 consensus_meta.cc:287] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.551s user 0.010s sys 0.000s
W20251023 02:05:35.763417 24792 consensus_meta.cc:287] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.592s user 0.008s sys 0.001s
I20251023 02:05:35.763803 24792 raft_consensus.cc:515] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.708735 24655 raft_consensus.cc:2804] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:35.765427 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:35.766070 22106 raft_consensus.cc:2468] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:35.766706 24655 raft_consensus.cc:493] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:35.767000 22106 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:35.593869 24725 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.144s
I20251023 02:05:35.601358 24864 raft_consensus.cc:515] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.769318 24864 leader_election.cc:290] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:35.744925 24641 consensus_queue.cc:237] T 4814f8f6136a4c1490329f260ced6c47 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.770936 24898 raft_consensus.cc:3055] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:35.771329 24898 raft_consensus.cc:740] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:35.745785 24439 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:35.748472 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:35.772459 22618 raft_consensus.cc:1240] T 816ef481cee84b259bc409b45a81baf1 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 LEADER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:35.603706 24292 raft_consensus.cc:515] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.772184 24898 consensus_queue.cc:260] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.749665 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
W20251023 02:05:35.774433 24945 log.cc:927] Time spent T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.318s user 0.002s sys 0.000s
I20251023 02:05:35.774631 22111 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:35.590507 24881 consensus_queue.cc:1048] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.135s
I20251023 02:05:35.776392 22102 raft_consensus.cc:2393] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
W20251023 02:05:35.782591 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.784826 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.739881 24892 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.786979 24790 raft_consensus.cc:3060] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.790118 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.790931 22624 raft_consensus.cc:2393] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 8: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:35.792201 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.793262 22622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 286) took 1972 ms. Trace:
I20251023 02:05:35.793642 22622 rpcz_store.cc:276] 1023 02:05:33.821506 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:33.824021 (+ 2515us) service_pool.cc:225] Handling call
1023 02:05:33.824452 (+ 431us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:33.824841 (+ 389us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:33.824883 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:33.825503 (+ 620us) log.cc:844] Serialized 26 byte log entry
1023 02:05:35.020281 (+1194778us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b180040b640. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5f63d55c 00007f9c5c9ce3bb 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9eabf2 00007f9c5f62f2bb 00007f9c5f634dc3 00007f9c5f635171
1023 02:05:35.020612 (+ 331us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:35.020662 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:35.020845 (+ 183us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:35.589763 (+568918us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:35.792964 (+203201us) raft_consensus.cc:1685] finished
1023 02:05:35.792985 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:35.793089 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10937,"raft.queue_time_us":55916,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":25733,"spinlock_wait_cycles":452883584,"thread_start_us":87727,"threads_started":1,"wal-append.queue_time_us":525719}
W20251023 02:05:35.796339 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 57940us
I20251023 02:05:35.798501 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:35.744933 22109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:35.740257 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:35.657049 24789 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.803445 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.804512 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:35.805328 22621 raft_consensus.cc:1240] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 11 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 10. Current term is 11. Ops: []
I20251023 02:05:35.811403 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.811980 21955 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:35.813876 21959 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 8: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:05:35.814142 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 103079us
I20251023 02:05:35.814399 24813 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:35.815627 22031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 307) took 2929 ms. Trace:
I20251023 02:05:35.792225 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:35.820011 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 43852.1us
I20251023 02:05:35.820281 21962 raft_consensus.cc:2393] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 8: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:35.814590 22110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1623) took 2161 ms. Trace:
I20251023 02:05:35.767165 24655 raft_consensus.cc:3060] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.820797 22110 rpcz_store.cc:276] 1023 02:05:33.653709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:34.352944 (+699235us) service_pool.cc:225] Handling call
1023 02:05:34.353204 (+ 260us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:34.973425 (+620221us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:35.814370 (+840945us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b54001ce3d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:35.814538 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1278711680}
I20251023 02:05:35.773975 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.812829 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.832739 22106 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:35.594450 24889 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.189s
I20251023 02:05:35.713282 24905 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
W20251023 02:05:35.840754 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 351227us
I20251023 02:05:35.742836 24909 raft_consensus.cc:3060] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.851084 24883 raft_consensus.cc:515] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:35.860131 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.856070 24897 raft_consensus.cc:493] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:35.842211 24921 log.cc:927] Time spent T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
W20251023 02:05:35.862334 24744 consensus_meta.cc:287] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.523s user 0.014s sys 0.000s
I20251023 02:05:35.863174 24292 leader_election.cc:290] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:35.864450 24892 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:35.862367 24514 consensus_meta.cc:287] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.541s user 0.013s sys 0.000s
I20251023 02:05:35.857301 22103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1627) took 1086 ms. Trace:
I20251023 02:05:35.773507 24898 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.816004 22031 rpcz_store.cc:276] 1023 02:05:32.886667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:33.938625 (+1051958us) service_pool.cc:225] Handling call
1023 02:05:34.234888 (+296263us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:35.233773 (+998885us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b18000c4ae0. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c658309a1 00007f9c6583d371 00007f9c5f63d55c 00007f9c5c9ce3bb 00007f9c657a9219 00007f9c5c9ea68b 00007f9c5c9e892f 00007f9c5c9e8445 00007f9c5fa90c00 00007f9c5fa9179b 00007f9c5fa9c1db
1023 02:05:35.233952 (+ 179us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:35.233972 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:35.234585 (+ 613us) log.cc:844] Serialized 52 byte log entry
1023 02:05:35.496705 (+262120us) raft_consensus.cc:1645] Marking committed up to 7
1023 02:05:35.496750 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:35.815187 (+318437us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b540036d654. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:35.815249 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:35.815290 (+ 41us) raft_consensus.cc:1685] finished
1023 02:05:35.815304 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:35.815472 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7192,"raft.queue_time_us":11335,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":11663,"spinlock_wait_cycles":505483008,"thread_start_us":290201,"threads_started":2,"wal-append.queue_time_us":324565}
I20251023 02:05:35.883617 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.870987 22622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.884407 22622 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:35.872486 21957 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:35.870419 24789 leader_election.cc:290] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:35.891170 24836 raft_consensus.cc:493] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:35.891527 24928 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:35.891925 24883 leader_election.cc:290] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:35.891644 24836 raft_consensus.cc:515] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.892755 24883 raft_consensus.cc:2804] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:35.878614 21962 raft_consensus.cc:2468] T 972d07041fbd4097854ef23c73f09c75 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 7.
I20251023 02:05:35.893193 24883 raft_consensus.cc:493] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:35.865455 24885 raft_consensus.cc:3055] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:35.893834 24885 raft_consensus.cc:740] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:35.894083 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.895076 21962 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 5.
I20251023 02:05:35.894867 24885 consensus_queue.cc:260] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.896121 24885 raft_consensus.cc:3060] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.896342 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.897972 21956 raft_consensus.cc:2410] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251023 02:05:35.893850 24836 leader_election.cc:290] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:35.905735 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 71926.2us
I20251023 02:05:35.907624 24849 consensus_queue.cc:1059] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251023 02:05:35.908519 24849 raft_consensus.cc:3055] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 LEADER]: Stepping down as leader of term 10
I20251023 02:05:35.908829 24849 raft_consensus.cc:740] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:35.909723 24831 raft_consensus.cc:3055] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:35.910100 24831 raft_consensus.cc:740] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:35.880357 24744 raft_consensus.cc:515] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.934368 22034 raft_consensus.cc:1240] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20251023 02:05:35.935451 24744 leader_election.cc:290] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:35.911139 21908 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d2ed2c541deb4c8eb9997f54a56f68d7, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:35.939631 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d2ed2c541deb4c8eb9997f54a56f68d7, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.911341 24543 raft_consensus.cc:3055] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Stepping down as leader of term 6
I20251023 02:05:35.941449 24543 raft_consensus.cc:740] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:35.942579 24543 consensus_queue.cc:260] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.944070 24543 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:35.881435 22103 rpcz_store.cc:276] 1023 02:05:34.770439 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:35.236649 (+466210us) service_pool.cc:225] Handling call
1023 02:05:35.236983 (+ 334us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:35.277059 (+ 40076us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:35.277089 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:35.292438 (+ 15349us) log.cc:844] Serialized 26 byte log entry
1023 02:05:35.700796 (+408358us) raft_consensus.cc:1645] Marking committed up to 5
1023 02:05:35.700859 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:35.700953 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:35.857060 (+156107us) raft_consensus.cc:1685] finished
1023 02:05:35.857088 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:35.857235 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70108,"raft.queue_time_us":44499,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":10186,"thread_start_us":344139,"threads_started":1,"wal-append.queue_time_us":441922}
I20251023 02:05:35.952729 21962 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 9.
I20251023 02:05:35.911461 22622 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20251023 02:05:35.876453 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.912773 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.955252 21955 raft_consensus.cc:2468] T 365d47bf062b4ebabb6d5f247e4e3e33 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:35.956621 22589 leader_election.cc:304] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:35.873765 24558 raft_consensus.cc:515] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.913656 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.914351 22624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.915046 22616 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.916745 22618 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:35.955458 22623 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:35.878463 24920 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.889308 24892 consensus_queue.cc:1059] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20251023 02:05:35.879442 24792 leader_election.cc:290] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 11 election: Requested vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:35.921622 24514 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:35.973338 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 377641us
I20251023 02:05:35.973903 24514 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.975222 22077 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:35.976099 24514 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:35.976398 24576 raft_consensus.cc:2749] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:35.976728 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:35.922658 22030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39012 (request call id 1336) took 1324 ms. Trace:
I20251023 02:05:35.922742 22002 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:35.892289 24928 raft_consensus.cc:515] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.977725 22030 rpcz_store.cc:276] 1023 02:05:34.598499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:35.040076 (+441577us) service_pool.cc:225] Handling call
1023 02:05:35.043255 (+ 3179us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:35.658552 (+615297us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:35.922397 (+263845us) spinlock_profiling.cc:243] Waited 553 ms on lock 0x7b540034a154. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:35.922573 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1271094016}
I20251023 02:05:35.925714 24883 raft_consensus.cc:3060] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:35.925925 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:35.926541 22620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:35.928190 24870 consensus_meta.cc:287] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.522s user 0.001s sys 0.009s
I20251023 02:05:35.928606 22111 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 10.
W20251023 02:05:35.927547 24798 consensus_meta.cc:287] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.560s user 0.007s sys 0.002s
I20251023 02:05:35.987808 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:35.930800 22028 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:35.931432 21958 raft_consensus.cc:2468] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
W20251023 02:05:35.932049 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:35.932416 22102 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251023 02:05:35.910360 22617 raft_consensus.cc:1275] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20251023 02:05:35.960820 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 72687.7us
I20251023 02:05:35.960666 24949 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:35.998232 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "422ac2af786e439185d0e762e06ef0a3" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:35.964250 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:35.999143 21962 raft_consensus.cc:2393] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 8: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:35.966729 24920 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:35.967489 24889 raft_consensus.cc:3055] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:35.968078 24779 consensus_queue.cc:1048] T 0d2df37149b145b89207c30e1a0360b0 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20251023 02:05:35.968761 24878 raft_consensus.cc:515] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } attrs { replace: true } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.969441 22029 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66"
dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
mode: GRACEFUL
new_leader_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
from {username='slave'} at 127.0.0.1:48448
I20251023 02:05:35.971321 21959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5d794166cc4fbb87c6b918fa26b592" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:36.006371 21959 raft_consensus.cc:3060] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:36.006503 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:35.971088 24831 consensus_queue.cc:260] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.972430 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:36.007119 21957 raft_consensus.cc:3060] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:35.977519 21956 raft_consensus.cc:2410] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251023 02:05:35.923525 24849 consensus_queue.cc:260] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:35.980599 22624 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:35.875895 24897 raft_consensus.cc:515] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:35.878473 24848 raft_consensus.cc:515] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:35.977753 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 391384us
I20251023 02:05:35.981981 22105 raft_consensus.cc:1240] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20251023 02:05:35.964498 24670 raft_consensus.cc:2804] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251023 02:05:36.003894 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.001086 22587 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:36.006009 22029 raft_consensus.cc:606] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Received request to transfer leadership to TS 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:36.074445 22029 raft_consensus.cc:614] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251023 02:05:35.999004 24876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.008s sys 0.008s
I20251023 02:05:36.011193 21955 raft_consensus.cc:2393] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 9: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:36.005878 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2df37149b145b89207c30e1a0360b0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:36.027925 22077 leader_election.cc:304] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:36.046872 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.005347 24543 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.075155 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:36.000353 24744 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:36.077111 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:36.109268 21962 raft_consensus.cc:2393] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 8: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:36.074957 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.010636 22619 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:36.079332 22028 raft_consensus.cc:2393] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 10: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
W20251023 02:05:36.081611 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 73199.3us
I20251023 02:05:36.003511 24889 raft_consensus.cc:740] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:36.093391 21956 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e in current term 8: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
W20251023 02:05:36.076431 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:36.093771 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 92828.6us
I20251023 02:05:36.093873 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:36.020116 24850 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.013s sys 0.007s
W20251023 02:05:36.122210 24850 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.667s user 0.014s sys 0.008s
I20251023 02:05:35.989554 24870 raft_consensus.cc:515] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.126040 22108 raft_consensus.cc:2368] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9.
I20251023 02:05:36.131515 22107 raft_consensus.cc:1275] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251023 02:05:36.134306 22102 raft_consensus.cc:1275] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251023 02:05:36.135735 22077 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:36.134321 20804 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251023 02:05:36.107506 24791 raft_consensus.cc:493] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.137670 22103 raft_consensus.cc:1240] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:36.138137 24791 raft_consensus.cc:515] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.140661 24791 leader_election.cc:290] T f62998a4a7ab4a4a878db25922f83a69 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:36.097061 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 171984us
I20251023 02:05:36.106163 24543 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.153029 22002 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:36.153564 24543 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.105998 24667 raft_consensus.cc:493] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:36.154522 24667 raft_consensus.cc:515] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.155125 24858 raft_consensus.cc:2764] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20251023 02:05:36.155956 24858 consensus_queue.cc:1048] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251023 02:05:36.156383 24667 leader_election.cc:290] T 00301d3b4a804bf19bad66c51fe19faa P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.156852 24965 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251023 02:05:36.157299 21926 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 76016.6us
I20251023 02:05:36.157536 24965 consensus_queue.cc:1059] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251023 02:05:36.098153 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.105974 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.159394 22035 raft_consensus.cc:2393] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 10: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:36.159979 24897 leader_election.cc:290] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.160418 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.160646 24897 consensus_queue.cc:1059] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20251023 02:05:36.106601 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:36.162204 22621 raft_consensus.cc:2393] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 8: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:36.093473 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:36.163373 22030 raft_consensus.cc:3060] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:36.106601 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.117063 22589 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:36.167313 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 50368.6us
I20251023 02:05:35.990531 24798 raft_consensus.cc:515] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.008652 24949 raft_consensus.cc:515] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:36.170565 24798 leader_election.cc:290] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:36.085628 24876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.008s sys 0.008s
I20251023 02:05:36.172104 24779 raft_consensus.cc:2804] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:36.172643 24779 raft_consensus.cc:493] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:36.173054 24779 raft_consensus.cc:3060] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:36.176303 20804 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251023 02:05:36.176617 24909 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:36.121598 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:36.157435 22002 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:36.121789 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:36.145138 24964 log.cc:927] Time spent T 5470b0fad25247c19157cd390de47f7f P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20251023 02:05:36.149291 24872 consensus_meta.cc:287] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.704s user 0.006s sys 0.006s
I20251023 02:05:36.149950 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:36.185935 24798 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.096547 22037 raft_consensus.cc:1240] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20251023 02:05:36.120729 24924 consensus_meta.cc:287] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.666s user 0.009s sys 0.000s
I20251023 02:05:36.151016 24591 raft_consensus.cc:2804] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:36.178053 20804 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251023 02:05:36.121286 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:36.207497 21962 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:36.125342 24870 leader_election.cc:290] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:36.180925 24889 consensus_queue.cc:260] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:36.183802 24878 leader_election.cc:290] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:36.185492 24848 leader_election.cc:290] T 1aaf7b9acded449f9017f47455026853 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:36.073886 24670 raft_consensus.cc:493] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:36.117960 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:36.195519 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:36.195850 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:36.196213 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:36.217712 22107 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:36.150214 24558 leader_election.cc:290] T de76561746ce42549f51431c3f40bed7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:36.198906 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.201640 21957 raft_consensus.cc:2468] T f932b6202f5b497e8fb358af3c5d3f66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 10.
I20251023 02:05:36.203297 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c29a39800fe46e9b0b85ea73ecc703f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.224037 22035 raft_consensus.cc:2468] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 4.
I20251023 02:05:36.204046 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.224923 21956 raft_consensus.cc:2468] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 10.
I20251023 02:05:36.225096 22031 raft_consensus.cc:2468] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
I20251023 02:05:36.204217 24798 raft_consensus.cc:515] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.209319 22036 raft_consensus.cc:3060] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:36.193939 24849 raft_consensus.cc:3060] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Advancing to term 11
I20251023 02:05:36.211290 24831 raft_consensus.cc:3060] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:36.234805 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.235121 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
W20251023 02:05:36.222105 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 64781.2us
I20251023 02:05:36.236953 21959 raft_consensus.cc:2468] T fe5d794166cc4fbb87c6b918fa26b592 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:36.227005 21960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c954cadc886d47b39ed2cf51f9ac59f7" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:36.217274 22034 raft_consensus.cc:2376] T 12db617f200e4104b5346432288ad306 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5. Re-sending same reply.
I20251023 02:05:36.212733 22030 raft_consensus.cc:2468] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
W20251023 02:05:36.232432 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.233767 24689 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:36.226609 24881 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.221952 24920 raft_consensus.cc:2804] T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:36.222254 24670 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:36.264124 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.200156 24655 raft_consensus.cc:515] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.242923 22002 leader_election.cc:304] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:36.243206 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:36.213828 24591 raft_consensus.cc:697] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:36.276988 21960 raft_consensus.cc:1240] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251023 02:05:36.264698 21957 raft_consensus.cc:1275] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251023 02:05:36.279505 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:36.280097 22617 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:36.280934 24798 leader_election.cc:290] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.282130 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:36.282477 24591 consensus_queue.cc:237] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:36.283358 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90dcf9b1f8874437a7cd406b2ace7eeb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.284300 22031 raft_consensus.cc:2410] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 9 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251023 02:05:36.285110 22031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d80086b174ba4cee80e75f95401994e3" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.283797 24779 raft_consensus.cc:515] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:36.289224 24779 leader_election.cc:290] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.275205 24792 consensus_queue.cc:1059] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
W20251023 02:05:36.291759 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 48955.1us
I20251023 02:05:36.247248 22590 leader_election.cc:304] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:36.294935 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 47786.9us
I20251023 02:05:36.245175 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:36.205760 24905 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.206350 24507 raft_consensus.cc:515] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:36.239070 24868 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.818s user 0.008s sys 0.047s
I20251023 02:05:36.268121 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97dae00618e34f58962a1dfbaa38fcdc" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:36.318795 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.264910 21955 raft_consensus.cc:2468] T d80086b174ba4cee80e75f95401994e3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 5.
I20251023 02:05:36.272187 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.272693 22030 raft_consensus.cc:2410] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251023 02:05:36.287453 24889 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
W20251023 02:05:36.246196 22001 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.244123 24689 raft_consensus.cc:493] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:36.308234 22002 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:36.312220 22036 raft_consensus.cc:2468] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
W20251023 02:05:36.313541 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.313967 22107 raft_consensus.cc:2468] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 10.
I20251023 02:05:36.291110 24779 raft_consensus.cc:3055] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 LEADER]: Stepping down as leader of term 9
I20251023 02:05:36.269644 24543 raft_consensus.cc:493] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ca1362b9d9b34073bf35b6f6c9d1159a)
I20251023 02:05:36.331796 21959 raft_consensus.cc:1275] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20251023 02:05:36.318459 24868 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.898s user 0.008s sys 0.047s
I20251023 02:05:36.264395 24848 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.078s
I20251023 02:05:36.336748 22104 raft_consensus.cc:1240] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:36.317757 24792 raft_consensus.cc:2804] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader election won for term 6
I20251023 02:05:36.267982 24881 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.314944 24885 raft_consensus.cc:2749] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251023 02:05:36.328362 24779 raft_consensus.cc:740] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
W20251023 02:05:36.391613 22076 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.015s sys 0.030s
I20251023 02:05:36.391737 22034 raft_consensus.cc:1275] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251023 02:05:36.392961 21961 raft_consensus.cc:1240] T e8a971ac884645548d12e7c83cc039bc P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 LEADER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
W20251023 02:05:36.393496 22076 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s user 0.015s sys 0.030s
I20251023 02:05:36.378692 22035 raft_consensus.cc:1275] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20251023 02:05:36.395727 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 72566.6us
I20251023 02:05:36.329525 24689 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:36.403821 22036 raft_consensus.cc:1275] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251023 02:05:36.399438 22000 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:36.419935 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.422164 22076 leader_election.cc:304] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:36.430632 21960 raft_consensus.cc:1275] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251023 02:05:36.441078 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:36.441943 22035 raft_consensus.cc:3060] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:36.393226 24346 raft_consensus.cc:2749] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251023 02:05:36.442305 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 134167us
I20251023 02:05:36.447068 22002 leader_election.cc:304] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:36.441263 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98f1206f4e204b72af762bd083d766fd" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:36.396592 24792 raft_consensus.cc:697] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 LEADER]: Becoming Leader. State: Replica: 1ab8b0b58147446d84d901b4ed2ff39e, State: Running, Role: LEADER
I20251023 02:05:36.449128 21959 raft_consensus.cc:1275] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20251023 02:05:36.443287 21956 raft_consensus.cc:1240] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 6. Current term is 7. Ops: []
W20251023 02:05:36.451196 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 51807.9us
I20251023 02:05:36.374840 24543 raft_consensus.cc:515] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:36.460772 22000 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:36.458236 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da2ca0f34944635a558b8678ecc206f" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.459565 21959 pending_rounds.cc:85] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14: Aborting all ops after (but not including) 3
I20251023 02:05:36.437363 24558 raft_consensus.cc:493] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.476994 22002 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:36.435006 24292 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.470113 21959 pending_rounds.cc:107] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20251023 02:05:36.436967 24909 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.482255 21959 raft_consensus.cc:2889] T 57e843357de442f3a618eac9934513ed P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251023 02:05:36.488714 22000 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
W20251023 02:05:36.483273 21928 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.448395 24958 raft_consensus.cc:493] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.485493 21956 raft_consensus.cc:1275] T c4c983d0648c4b649e08012b7932a15a P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251023 02:05:36.394277 24968 consensus_queue.cc:1059] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.225s
I20251023 02:05:36.465622 24872 raft_consensus.cc:2804] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election won for term 9
W20251023 02:05:36.498618 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 76603.8us
I20251023 02:05:36.504432 22076 leader_election.cc:304] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:36.507418 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f745a2e5ae9f4a829fa01ad2bce79767" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:36.515547 21962 raft_consensus.cc:2468] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:36.520535 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:36.520697 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 398603us
W20251023 02:05:36.521560 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:36.522121 22075 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.523240 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:36.523811 22107 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:36.497438 24876 raft_consensus.cc:493] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.526250 22076 leader_election.cc:304] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:36.526957 22076 leader_election.cc:304] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:36.497555 24813 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.051s sys 0.090s
I20251023 02:05:36.527532 22034 raft_consensus.cc:2410] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251023 02:05:36.502261 24323 raft_consensus.cc:3060] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 3 FOLLOWER]: Advancing to term 4
I20251023 02:05:36.502712 24885 raft_consensus.cc:2749] T 3c29a39800fe46e9b0b85ea73ecc703f P 818907d63f1f4d84ba267ebe6ee837e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:36.499665 24346 raft_consensus.cc:2804] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:36.533692 22621 raft_consensus.cc:1240] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 5. Current term is 6. Ops: []
I20251023 02:05:36.533563 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12db617f200e4104b5346432288ad306" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:36.541565 22076 leader_election.cc:304] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:36.497272 24292 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:36.529716 24813 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s user 0.052s sys 0.090s
I20251023 02:05:36.565079 24813 raft_consensus.cc:3055] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Stepping down as leader of term 5
I20251023 02:05:36.566911 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
W20251023 02:05:36.567178 24883 consensus_meta.cc:287] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.580s user 0.011s sys 0.000s
I20251023 02:05:36.567751 24883 raft_consensus.cc:515] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:36.569747 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 104243us
I20251023 02:05:36.516171 24847 raft_consensus.cc:2749] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:36.523026 24872 raft_consensus.cc:697] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:36.570490 22002 leader_election.cc:304] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:36.571712 24872 consensus_queue.cc:237] T 8058bbefb0b74ae5949474c893a6947a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.494727 24779 consensus_queue.cc:260] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:36.574227 24779 raft_consensus.cc:3060] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:36.574910 24972 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.575621 24972 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.500672 24558 raft_consensus.cc:515] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.527833 24798 raft_consensus.cc:3055] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Stepping down as leader of term 8
I20251023 02:05:36.546324 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:36.542300 24905 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:36.547863 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:36.599591 22035 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:36.600080 22036 raft_consensus.cc:2393] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 7: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:36.553884 22033 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 320) took 1116 ms. Trace:
I20251023 02:05:36.600477 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.553689 24966 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20251023 02:05:36.534489 24924 raft_consensus.cc:2804] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251023 02:05:36.543893 24870 raft_consensus.cc:2804] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251023 02:05:36.538344 24655 leader_election.cc:290] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:36.562426 24928 leader_election.cc:290] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.562978 24963 consensus_queue.cc:1048] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.019s
I20251023 02:05:36.563855 22108 raft_consensus.cc:2468] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:36.568179 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:36.613410 22103 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 9.
I20251023 02:05:36.588999 24847 raft_consensus.cc:2749] T d2ed2c541deb4c8eb9997f54a56f68d7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20251023 02:05:36.588671 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 310963us
W20251023 02:05:36.616467 22589 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251023 02:05:36.617116 22590 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.549356 24292 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:36.578336 24813 raft_consensus.cc:740] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 02c53ba06cc3445a8534a84aa76b1c14, State: Running, Role: LEADER
I20251023 02:05:36.553766 24792 consensus_queue.cc:237] T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:36.533170 24735 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:36.542605 24507 leader_election.cc:290] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:36.531193 24876 raft_consensus.cc:515] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.631454 24876 leader_election.cc:290] T 4a43ae59c3e347fd86371312be59bb23 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.632438 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:36.590981 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:36.580000 24858 consensus_queue.cc:1059] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:05:36.572664 22002 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:36.586553 22037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55770 (request call id 318) took 1188 ms. Trace:
I20251023 02:05:36.634065 22037 rpcz_store.cc:276] 1023 02:05:35.398437 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:35.775372 (+376935us) service_pool.cc:225] Handling call
1023 02:05:35.775649 (+ 277us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:36.200595 (+424946us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:36.586286 (+385691us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b540040d3d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:36.586490 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":737356672}
I20251023 02:05:36.635188 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3bb66d66a264e8abdf220531328f12b" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.635864 22037 raft_consensus.cc:2393] T d3bb66d66a264e8abdf220531328f12b P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 9: Already voted for candidate 818907d63f1f4d84ba267ebe6ee837e0 in this term.
I20251023 02:05:36.636520 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.636612 22111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:36.596222 22107 raft_consensus.cc:2468] T 57e843357de442f3a618eac9934513ed P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:36.503077 24958 raft_consensus.cc:515] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:36.637885 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:36.638494 22107 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 9.
I20251023 02:05:36.638988 24958 leader_election.cc:290] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.604929 22587 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:36.640775 22107 raft_consensus.cc:1240] T 97dae00618e34f58962a1dfbaa38fcdc P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Rejecting Update request from peer 1ab8b0b58147446d84d901b4ed2ff39e for earlier term 9. Current term is 10. Ops: []
I20251023 02:05:36.600617 22033 rpcz_store.cc:276] 1023 02:05:35.437953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:35.930775 (+492822us) service_pool.cc:225] Handling call
1023 02:05:35.931121 (+ 346us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:35.931314 (+ 193us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:35.931359 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:35.931841 (+ 482us) log.cc:844] Serialized 26 byte log entry
1023 02:05:36.393992 (+462151us) raft_consensus.cc:1645] Marking committed up to 8
1023 02:05:36.394042 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:36.394132 (+ 90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:36.553654 (+159522us) raft_consensus.cc:1685] finished
1023 02:05:36.553675 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:36.553825 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":461757,"threads_started":1,"wal-append.queue_time_us":593928}
I20251023 02:05:36.539566 24346 raft_consensus.cc:493] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:36.642081 24346 raft_consensus.cc:3060] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:36.643556 21962 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:36.530177 24576 raft_consensus.cc:2804] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election won for term 9
I20251023 02:05:36.645850 24677 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251023 02:05:36.646664 22033 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "972d07041fbd4097854ef23c73f09c75" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.595894 24798 raft_consensus.cc:740] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:36.607024 24924 raft_consensus.cc:493] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:36.648177 24924 raft_consensus.cc:3060] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Advancing to term 11
I20251023 02:05:36.647020 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a971ac884645548d12e7c83cc039bc" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:36.626072 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.626205 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:36.626710 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:36.608733 24870 raft_consensus.cc:493] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:36.633602 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:36.645898 24576 raft_consensus.cc:697] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:36.596163 24850 raft_consensus.cc:2804] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:36.608836 24744 raft_consensus.cc:2749] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251023 02:05:36.657956 22077 leader_election.cc:336] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645): Timed out: RequestConsensusVote RPC to 127.20.81.2:40645 timed out after 1.660s (SENT)
I20251023 02:05:36.661648 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f932b6202f5b497e8fb358af3c5d3f66" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:36.661764 22077 leader_election.cc:304] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:36.650264 22037 raft_consensus.cc:2410] T e8a971ac884645548d12e7c83cc039bc P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251023 02:05:36.650974 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e923b66ccf04dd8985bbfe31adbf0b4" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.637405 22111 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
W20251023 02:05:36.654021 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 81430.9us
I20251023 02:05:36.599146 24627 consensus_queue.cc:1059] T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.024s
I20251023 02:05:36.606901 24909 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20251023 02:05:36.683104 22103 raft_consensus.cc:1275] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251023 02:05:36.683135 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:36.619949 24881 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.664472 22587 leader_election.cc:304] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:36.696035 22029 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:36.685317 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "864751dbbbb748769206d89ccd8cb70e" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:36.701727 21962 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:36.702385 22037 raft_consensus.cc:2393] T 864751dbbbb748769206d89ccd8cb70e P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 in current term 8: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:36.704260 22102 raft_consensus.cc:2410] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ab8b0b58147446d84d901b4ed2ff39e for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20251023 02:05:36.707027 24558 leader_election.cc:290] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:36.706805 24576 consensus_queue.cc:237] T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.708122 24958 raft_consensus.cc:2749] T 66c8e1c6015d47879ab7927548ad3b02 P 1ab8b0b58147446d84d901b4ed2ff39e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20251023 02:05:36.709110 22110 raft_consensus.cc:1275] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251023 02:05:36.709436 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.719475 22035 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 9.
I20251023 02:05:36.681267 24735 raft_consensus.cc:2804] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Leader election won for term 10
I20251023 02:05:36.701148 22036 raft_consensus.cc:2468] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 818907d63f1f4d84ba267ebe6ee837e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 5.
I20251023 02:05:36.661310 24972 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:36.711764 22002 leader_election.cc:304] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:36.683740 24883 leader_election.cc:290] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:36.725890 22076 leader_election.cc:304] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:36.703276 24677 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.731177 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:36.732681 21908 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f09d991e8bee44c98b35a4bd632685bf, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:36.775236 21908 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f09d991e8bee44c98b35a4bd632685bf, TS ca1362b9d9b34073bf35b6f6c9d1159a -> TS 1ab8b0b58147446d84d901b4ed2ff39e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:36.775650 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 49889.4us
I20251023 02:05:36.778741 22103 raft_consensus.cc:1275] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251023 02:05:36.780565 22621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
W20251023 02:05:36.782124 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 124216us
I20251023 02:05:36.782063 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4c983d0648c4b649e08012b7932a15a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.662328 24870 raft_consensus.cc:3060] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Advancing to term 6
I20251023 02:05:36.726438 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:36.751384 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.705123 24507 raft_consensus.cc:2749] T 12db617f200e4104b5346432288ad306 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251023 02:05:36.787708 21928 leader_election.cc:304] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:36.788506 21962 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:36.705674 24850 raft_consensus.cc:697] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:36.789225 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00301d3b4a804bf19bad66c51fe19faa" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.791360 22108 raft_consensus.cc:1240] T bd7f335f5df946c481e05b1a2a95f182 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:36.792366 22037 raft_consensus.cc:2468] T c4c983d0648c4b649e08012b7932a15a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 9.
I20251023 02:05:36.794353 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40b987b2b0ed4294b7ccf057ea33c0a0" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
W20251023 02:05:36.796288 22587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 131924us
W20251023 02:05:36.798298 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 86655.3us
I20251023 02:05:36.724895 24949 leader_election.cc:290] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.797317 22037 raft_consensus.cc:2410] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251023 02:05:36.807665 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8058bbefb0b74ae5949474c893a6947a" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.802736 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.803330 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a43ae59c3e347fd86371312be59bb23" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:36.725636 24543 leader_election.cc:290] T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
W20251023 02:05:36.810920 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 322344us
I20251023 02:05:36.811883 22000 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:36.783298 24735 raft_consensus.cc:697] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:36.818490 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.787073 24972 raft_consensus.cc:2804] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election won for term 9
I20251023 02:05:36.792325 24744 raft_consensus.cc:2749] T bd7f335f5df946c481e05b1a2a95f182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:36.825729 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0a3abec901e4c2eba45233d075d9263" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:36.828562 22108 raft_consensus.cc:2468] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 7.
I20251023 02:05:36.829808 22587 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:36.832913 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f62998a4a7ab4a4a878db25922f83a69" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:36.834228 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 46634.4us
I20251023 02:05:36.774161 24881 consensus_queue.cc:1048] T 4a43ae59c3e347fd86371312be59bb23 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20251023 02:05:36.809765 24627 raft_consensus.cc:2749] T 98f1206f4e204b72af762bd083d766fd P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251023 02:05:36.851778 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:36.858232 22036 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 9.
I20251023 02:05:36.835355 22110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:36.795185 24958 consensus_queue.cc:1048] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20251023 02:05:36.787595 24677 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:36.865787 22103 raft_consensus.cc:1275] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251023 02:05:36.862282 22035 raft_consensus.cc:2468] T f62998a4a7ab4a4a878db25922f83a69 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 8.
I20251023 02:05:36.862340 21928 leader_election.cc:304] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:36.870214 21962 raft_consensus.cc:2468] T 93b201674548485eaa29b78067f74a66 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:36.871403 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:36.872099 21962 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Advancing to term 9
W20251023 02:05:36.872985 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 61186.7us
I20251023 02:05:36.808947 24847 consensus_queue.cc:1048] T f745a2e5ae9f4a829fa01ad2bce79767 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
W20251023 02:05:36.855211 24670 consensus_meta.cc:287] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.561s user 0.010s sys 0.001s
I20251023 02:05:36.877240 22076 leader_election.cc:304] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:36.844151 24543 raft_consensus.cc:493] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.852263 24972 raft_consensus.cc:697] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
W20251023 02:05:36.893175 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:36.894404 22035 raft_consensus.cc:1240] T 1376a6af91b24e46859bf630618fcec3 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 7. Current term is 8. Ops: []
W20251023 02:05:36.898061 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 63011.7us
I20251023 02:05:36.898947 21928 leader_election.cc:304] T f745a2e5ae9f4a829fa01ad2bce79767 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:36.865463 24744 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.824640 24831 consensus_queue.cc:1059] T 98f1206f4e204b72af762bd083d766fd P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
W20251023 02:05:36.911464 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 49540.4us
I20251023 02:05:36.851758 24885 consensus_queue.cc:1048] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20251023 02:05:36.864598 24968 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.929509 21962 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:36.930961 22076 leader_election.cc:304] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
W20251023 02:05:36.913542 22617 consensus_meta.cc:287] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.633s user 0.006s sys 0.006s
I20251023 02:05:36.894564 24836 raft_consensus.cc:2749] T 2da2ca0f34944635a558b8678ecc206f P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251023 02:05:36.928537 22075 leader_election.cc:304] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:36.925287 24849 consensus_meta.cc:287] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.687s user 0.007s sys 0.004s
I20251023 02:05:36.937719 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:36.940913 22617 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 1ab8b0b58147446d84d901b4ed2ff39e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 10.
I20251023 02:05:36.946231 21955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36488 (request call id 234) took 1108 ms. Trace:
W20251023 02:05:36.946627 24979 log.cc:927] Time spent T 422ac2af786e439185d0e762e06ef0a3 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20251023 02:05:36.923828 24507 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.886397 24875 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:36.933928 24740 raft_consensus.cc:493] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:36.882272 24689 raft_consensus.cc:515] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.936726 24968 raft_consensus.cc:515] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.961990 21957 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1711) took 1419 ms. Trace:
I20251023 02:05:36.962423 24689 leader_election.cc:290] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:36.962587 24968 leader_election.cc:290] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
W20251023 02:05:36.963487 21958 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53734 (request call id 1707) took 1492 ms (client timeout 1975 ms). Trace:
I20251023 02:05:36.883895 24670 raft_consensus.cc:515] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
W20251023 02:05:36.963861 21958 rpcz_store.cc:269] 1023 02:05:35.471031 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:35.710260 (+239229us) service_pool.cc:225] Handling call
1023 02:05:36.963334 (+1253074us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54000c3d54. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:36.963436 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":436525312}
I20251023 02:05:36.965008 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:36.965677 24670 leader_election.cc:290] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 10 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:36.939514 24872 raft_consensus.cc:2804] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election won for term 9
I20251023 02:05:36.968639 24798 consensus_queue.cc:260] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.969897 22031 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:36.954607 22075 leader_election.cc:304] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:36.894119 24883 raft_consensus.cc:2804] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election won for term 9
I20251023 02:05:36.946681 21955 rpcz_store.cc:276] 1023 02:05:35.837930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:36.330132 (+492202us) service_pool.cc:225] Handling call
1023 02:05:36.346494 (+ 16362us) raft_consensus.cc:1471] Updating replica for 2 ops
1023 02:05:36.393780 (+ 47286us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:36.393813 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1023 02:05:36.435195 (+ 41382us) log.cc:844] Serialized 52 byte log entry
1023 02:05:36.826809 (+391614us) raft_consensus.cc:1645] Marking committed up to 7
1023 02:05:36.826870 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:36.826960 (+ 90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:36.945984 (+119024us) raft_consensus.cc:1685] finished
1023 02:05:36.946005 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:36.946155 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45855,"raft.queue_time_us":50363,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":16865,"thread_start_us":324104,"threads_started":1,"wal-append.queue_time_us":449773}
I20251023 02:05:36.971567 22032 raft_consensus.cc:2410] T 66c8e1c6015d47879ab7927548ad3b02 P 818907d63f1f4d84ba267ebe6ee837e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
W20251023 02:05:37.000137 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 45612.2us
I20251023 02:05:36.962355 21957 rpcz_store.cc:276] 1023 02:05:35.542435 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:36.302072 (+759637us) service_pool.cc:225] Handling call
1023 02:05:36.330479 (+ 28407us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:36.390420 (+ 59941us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:36.390467 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:36.403419 (+ 12952us) log.cc:844] Serialized 26 byte log entry
1023 02:05:36.836304 (+432885us) raft_consensus.cc:1645] Marking committed up to 3
1023 02:05:36.836359 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:36.837665 (+ 1306us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:36.961745 (+124080us) raft_consensus.cc:1685] finished
1023 02:05:36.961768 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:36.961931 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24557,"raft.queue_time_us":43119,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":15926,"thread_start_us":367721,"threads_started":1,"wal-append.queue_time_us":503688}
I20251023 02:05:37.000869 21959 raft_consensus.cc:1275] T 1376a6af91b24e46859bf630618fcec3 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Refusing update from remote peer ca1362b9d9b34073bf35b6f6c9d1159a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251023 02:05:36.916005 24543 raft_consensus.cc:515] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:37.008543 22000 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.3:33899, user_credentials={real_user=slave}} blocked reactor thread for 125464us
I20251023 02:05:37.010473 22075 leader_election.cc:304] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:36.913832 24996 consensus_queue.cc:1059] T 1aaf7b9acded449f9017f47455026853 P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20251023 02:05:37.037031 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:37.046406 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf578f7b0cdb4188bb43d676b1c64e28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:37.046476 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:37.048838 21957 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:36.952481 24875 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:36.932732 24744 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.053766 21955 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:36.999495 24558 raft_consensus.cc:2749] T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251023 02:05:37.058081 21928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 159256us
W20251023 02:05:37.058219 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 132914us
I20251023 02:05:37.009334 24798 raft_consensus.cc:3060] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:37.005391 24872 raft_consensus.cc:697] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:37.036747 24655 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:37.037514 24292 raft_consensus.cc:2749] T 422ac2af786e439185d0e762e06ef0a3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:37.069072 22000 leader_election.cc:304] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:37.069944 21958 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
W20251023 02:05:37.074512 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 73734.1us
I20251023 02:05:37.074954 21958 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:37.038075 24883 raft_consensus.cc:697] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:37.075271 22075 leader_election.cc:304] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:37.074569 22000 leader_election.cc:400] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Vote denied by peer ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899) with higher term. Message: Invalid argument: T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 8. Current term is 9.
I20251023 02:05:37.082134 22000 leader_election.cc:403] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20251023 02:05:37.009510 24968 consensus_queue.cc:1048] T c0a3abec901e4c2eba45233d075d9263 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20251023 02:05:36.970804 24740 raft_consensus.cc:515] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.971947 24507 raft_consensus.cc:515] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:36.999527 24881 consensus_queue.cc:1048] T 57e843357de442f3a618eac9934513ed P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20251023 02:05:37.077788 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:36.932550 24850 consensus_queue.cc:237] T f09d991e8bee44c98b35a4bd632685bf P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.2, Last appended by leader: 2, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.109678 22030 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2b6e2b19ddc453290a76212731a2f28" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:37.079909 21928 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:37.055656 24875 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:37.089215 21957 raft_consensus.cc:2468] T 6dddaeefc3d449a0ba9b3e6695e37507 P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 10.
I20251023 02:05:36.934656 24735 consensus_queue.cc:237] T f932b6202f5b497e8fb358af3c5d3f66 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.093755 22077 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:37.094846 22075 leader_election.cc:304] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:37.113194 22103 raft_consensus.cc:1275] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20251023 02:05:37.114255 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:37.114516 22030 raft_consensus.cc:1240] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 6. Current term is 7. Ops: []
I20251023 02:05:37.115387 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ba6e9fe4bb44e559397893fa9d899db" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:37.115988 22103 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:37.078836 24878 raft_consensus.cc:2804] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:37.077030 24655 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:37.112092 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48cb01caedc4f359126f3d85087d709" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:37.127449 21957 raft_consensus.cc:3060] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:37.128518 22077 leader_election.cc:304] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
W20251023 02:05:37.130421 22590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 45364.8us
I20251023 02:05:37.116767 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:37.098207 21962 raft_consensus.cc:2468] T 7ba6e9fe4bb44e559397893fa9d899db P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:37.095103 24963 raft_consensus.cc:2749] T 774c010489aa4775bb6d636926900529 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251023 02:05:37.125928 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:37.135104 22107 raft_consensus.cc:2393] T 4814f8f6136a4c1490329f260ced6c47 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 7: Already voted for candidate 02c53ba06cc3445a8534a84aa76b1c14 in this term.
I20251023 02:05:37.126350 22108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dddaeefc3d449a0ba9b3e6695e37507" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:37.135779 22077 leader_election.cc:304] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14
I20251023 02:05:37.076100 24928 consensus_queue.cc:1059] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20251023 02:05:37.097677 24990 raft_consensus.cc:2804] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Leader election won for term 10
W20251023 02:05:37.137641 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:37.138800 22103 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:37.120254 24847 raft_consensus.cc:3060] T 93b201674548485eaa29b78067f74a66 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:37.120268 24949 raft_consensus.cc:2804] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251023 02:05:37.146600 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "365d47bf062b4ebabb6d5f247e4e3e33" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:37.148102 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:37.132995 24655 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
W20251023 02:05:37.135274 24323 consensus_meta.cc:287] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14: Time spent flushing consensus metadata: real 0.598s user 0.010s sys 0.000s
I20251023 02:05:37.149731 24928 consensus_queue.cc:1243] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer ca1362b9d9b34073bf35b6f6c9d1159a log is divergent from this leader: its last log entry 6.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251023 02:05:37.156919 22105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39646 (request call id 1641) took 2161 ms. Trace:
I20251023 02:05:37.152902 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4814f8f6136a4c1490329f260ced6c47" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:37.157284 22105 rpcz_store.cc:276] 1023 02:05:34.995067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:35.365641 (+370574us) service_pool.cc:225] Handling call
1023 02:05:35.389022 (+ 23381us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:36.030090 (+641068us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:37.156690 (+1126600us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b54001cf7d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:37.156864 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1363326720}
I20251023 02:05:37.068387 24972 consensus_queue.cc:237] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } }
I20251023 02:05:37.153769 22107 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:37.134912 24966 raft_consensus.cc:2804] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251023 02:05:37.160306 22587 leader_election.cc:304] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:37.159008 21957 raft_consensus.cc:2468] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:37.161285 22103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:37.136713 24292 raft_consensus.cc:3060] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:37.161931 22103 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:37.138367 24677 leader_election.cc:290] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:37.165961 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:37.170977 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "496e171acd9c4d6d9dab2ab594c06e57" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:37.173349 21956 raft_consensus.cc:2468] T f18823a673d148e892e11c43e6430242 P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:37.174410 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7600022898784926af7a5289ce03bc3a" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:37.174929 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffe0e348fc464e36be33082ddcdba159" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:37.175050 21956 raft_consensus.cc:2468] T 7600022898784926af7a5289ce03bc3a P 02c53ba06cc3445a8534a84aa76b1c14 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 9.
I20251023 02:05:37.175565 21962 raft_consensus.cc:2468] T ffe0e348fc464e36be33082ddcdba159 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 6.
I20251023 02:05:37.176374 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc4e5ba4d0d4809a19f394786d00288" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14"
I20251023 02:05:37.176909 21962 raft_consensus.cc:3060] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:37.176924 22076 leader_election.cc:304] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:37.178897 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:37.180333 21956 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818907d63f1f4d84ba267ebe6ee837e0 in term 8.
I20251023 02:05:37.181746 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:37.183521 21956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:37.187086 22105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aaf7b9acded449f9017f47455026853" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:37.188771 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 9 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:37.189891 22107 raft_consensus.cc:2410] T c0594e36e0824e13965498e412eee3fb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251023 02:05:37.190572 24883 consensus_queue.cc:237] T 93b201674548485eaa29b78067f74a66 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.159073 24963 raft_consensus.cc:2749] T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251023 02:05:37.194968 21957 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:37.196110 21957 raft_consensus.cc:2468] T edc3c9b5058946bf8f0da8fe69900291 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:37.161671 24966 raft_consensus.cc:493] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 02c53ba06cc3445a8534a84aa76b1c14)
I20251023 02:05:37.203967 21962 raft_consensus.cc:2468] T 0dc4e5ba4d0d4809a19f394786d00288 P 02c53ba06cc3445a8534a84aa76b1c14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7.
W20251023 02:05:37.208273 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 100950us
I20251023 02:05:37.231889 22105 raft_consensus.cc:1240] T 496e171acd9c4d6d9dab2ab594c06e57 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:37.188020 24990 raft_consensus.cc:697] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:37.199821 24892 raft_consensus.cc:2804] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251023 02:05:37.241935 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 106290us
I20251023 02:05:37.244704 22077 leader_election.cc:304] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
W20251023 02:05:37.247757 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 153003us
I20251023 02:05:37.207463 24966 raft_consensus.cc:3060] T 3c29a39800fe46e9b0b85ea73ecc703f P ca1362b9d9b34073bf35b6f6c9d1159a [term 4 FOLLOWER]: Advancing to term 5
I20251023 02:05:37.192759 24677 consensus_queue.cc:1048] T 864751dbbbb748769206d89ccd8cb70e P 1ab8b0b58147446d84d901b4ed2ff39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20251023 02:05:37.164916 24872 consensus_queue.cc:237] T 6869bc9cdea241aaa6d86d46a1b5bcfb P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.234457 21962 raft_consensus.cc:1240] T a48cb01caedc4f359126f3d85087d709 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: [6.4-7.5]
W20251023 02:05:37.272758 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 95959.7us
I20251023 02:05:37.231788 24968 raft_consensus.cc:3055] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:37.245945 24892 raft_consensus.cc:493] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:37.273452 22589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 181324us
I20251023 02:05:37.276099 22076 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:37.295840 22617 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 10 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 9. Current term is 10. Ops: []
W20251023 02:05:37.296442 24346 consensus_meta.cc:287] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0: Time spent flushing consensus metadata: real 0.654s user 0.008s sys 0.001s
W20251023 02:05:37.274824 24779 consensus_meta.cc:287] T 97dae00618e34f58962a1dfbaa38fcdc P 1ab8b0b58147446d84d901b4ed2ff39e: Time spent flushing consensus metadata: real 0.700s user 0.010s sys 0.001s
I20251023 02:05:37.297474 22028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1354) took 1281 ms. Trace:
I20251023 02:05:37.294108 22589 leader_election.cc:304] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:37.275687 21962 raft_consensus.cc:1240] T f09d991e8bee44c98b35a4bd632685bf P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:37.277385 22030 raft_consensus.cc:1275] T 74071b399ecc41808f0ac84dc836af4c P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Refusing update from remote peer 02c53ba06cc3445a8534a84aa76b1c14: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251023 02:05:37.311285 22028 rpcz_store.cc:276] 1023 02:05:35.999594 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:36.003649 (+ 4055us) service_pool.cc:225] Handling call
1023 02:05:37.281231 (+1277582us) spinlock_profiling.cc:243] Waited 4.84 ms on lock 0x7b540001d1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:37.281351 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11131264}
I20251023 02:05:37.248220 24744 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:37.304230 21955 raft_consensus.cc:2468] T 0325fa135bce480eaaa5faf74daf54f8 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:37.252415 24792 raft_consensus.cc:2804] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251023 02:05:37.318369 21955 raft_consensus.cc:1275] T 5470b0fad25247c19157cd390de47f7f P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Refusing update from remote peer 818907d63f1f4d84ba267ebe6ee837e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20251023 02:05:37.320395 21962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ead1daa70a847959916a302d8540676" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 2 } } ignore_live_leader: false dest_uuid: "02c53ba06cc3445a8534a84aa76b1c14" is_pre_election: true
I20251023 02:05:37.322322 22589 leader_election.cc:304] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:37.324365 22589 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ab8b0b58147446d84d901b4ed2ff39e; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:37.296867 24346 raft_consensus.cc:515] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:37.325989 24996 raft_consensus.cc:2749] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251023 02:05:37.326128 21962 raft_consensus.cc:2468] T 6ead1daa70a847959916a302d8540676 P 02c53ba06cc3445a8534a84aa76b1c14 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:37.326783 24996 raft_consensus.cc:2749] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20251023 02:05:37.326285 24999 log.cc:927] Time spent T 0d2df37149b145b89207c30e1a0360b0 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
I20251023 02:05:37.327764 24346 leader_election.cc:290] T 774c010489aa4775bb6d636926900529 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Requested vote from peers ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899), 1ab8b0b58147446d84d901b4ed2ff39e (127.20.81.4:33255)
I20251023 02:05:37.254607 24543 leader_election.cc:290] T d2ed2c541deb4c8eb9997f54a56f68d7 P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:37.327412 24996 raft_consensus.cc:2749] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251023 02:05:37.332373 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 87888.6us
I20251023 02:05:37.333431 22077 leader_election.cc:304] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:37.336668 22077 leader_election.cc:304] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:37.283514 24968 raft_consensus.cc:740] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
W20251023 02:05:37.342010 22001 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 43504.3us
W20251023 02:05:37.342113 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 105218us
I20251023 02:05:37.346050 22002 leader_election.cc:304] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 1ab8b0b58147446d84d901b4ed2ff39e, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:37.310827 24892 raft_consensus.cc:3060] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:37.319921 24576 raft_consensus.cc:2804] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251023 02:05:37.354049 24576 raft_consensus.cc:493] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:37.347263 24881 consensus_queue.cc:1059] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20251023 02:05:37.354504 24576 raft_consensus.cc:3060] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:37.358057 21954 raft_consensus.cc:1275] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Refusing update from remote peer 1ab8b0b58147446d84d901b4ed2ff39e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20251023 02:05:37.359472 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 83435.4us
I20251023 02:05:37.361331 22076 leader_election.cc:304] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:37.321102 24792 raft_consensus.cc:493] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:37.319797 24831 raft_consensus.cc:493] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:37.315456 24507 leader_election.cc:290] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
W20251023 02:05:37.374308 22075 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.81.4:33255, user_credentials={real_user=slave}} blocked reactor thread for 125856us
I20251023 02:05:37.361088 24876 raft_consensus.cc:493] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 02:05:37.384635 21925 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:37.385068 24876 raft_consensus.cc:515] T 0d2df37149b145b89207c30e1a0360b0 P 02c53ba06cc3445a8534a84aa76b1c14 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:37.385514 21925 leader_election.cc:304] T 864751dbbbb748769206d89ccd8cb70e P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14; no voters: 1ab8b0b58147446d84d901b4ed2ff39e, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:37.387385 22028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93b201674548485eaa29b78067f74a66" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:37.389873 22002 leader_election.cc:304] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
W20251023 02:05:37.391098 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 54537.2us
W20251023 02:05:37.420818 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 46784.3us
I20251023 02:05:37.373344 24792 raft_consensus.cc:3060] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 6 FOLLOWER]: Advancing to term 7
I20251023 02:05:37.421880 22002 leader_election.cc:304] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters: ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:37.422199 22077 leader_election.cc:304] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0
I20251023 02:05:37.326889 24990 consensus_queue.cc:237] T 97dae00618e34f58962a1dfbaa38fcdc P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:37.422735 22000 leader_election.cc:304] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 818907d63f1f4d84ba267ebe6ee837e0; no voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a
I20251023 02:05:37.387115 25003 raft_consensus.cc:2804] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 FOLLOWER]: Leader election won for term 10
I20251023 02:05:37.446281 22002 leader_election.cc:304] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:37.449200 22107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18823a673d148e892e11c43e6430242" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:37.390918 24507 raft_consensus.cc:2749] T 4a43ae59c3e347fd86371312be59bb23 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251023 02:05:37.457216 22621 raft_consensus.cc:1240] T cf578f7b0cdb4188bb43d676b1c64e28 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: []
W20251023 02:05:37.422787 24924 consensus_meta.cc:287] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.774s user 0.010s sys 0.001s
I20251023 02:05:37.464926 22037 raft_consensus.cc:2410] T 8058bbefb0b74ae5949474c893a6947a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c53ba06cc3445a8534a84aa76b1c14 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20251023 02:05:37.468657 22029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1360) took 1393 ms. Trace:
I20251023 02:05:37.475612 22029 rpcz_store.cc:276] 1023 02:05:36.075775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:36.097825 (+ 22050us) service_pool.cc:225] Handling call
1023 02:05:37.468502 (+1370677us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b540034b554. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:37.468604 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1235456768}
I20251023 02:05:37.469676 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6869bc9cdea241aaa6d86d46a1b5bcfb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:37.477000 22034 raft_consensus.cc:3060] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:37.461114 22002 leader_election.cc:304] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, 818907d63f1f4d84ba267ebe6ee837e0; no voters:
I20251023 02:05:37.420091 22617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e"
I20251023 02:05:37.442790 24346 raft_consensus.cc:2804] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20251023 02:05:37.453500 24870 consensus_meta.cc:287] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a: Time spent flushing consensus metadata: real 0.667s user 0.005s sys 0.006s
I20251023 02:05:37.450855 25013 raft_consensus.cc:2749] T 4814f8f6136a4c1490329f260ced6c47 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20251023 02:05:37.471745 25004 log.cc:927] Time spent T 5e923b66ccf04dd8985bbfe31adbf0b4 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.320s user 0.000s sys 0.002s
W20251023 02:05:37.484714 25027 log.cc:927] Time spent T fb1e5f57f70948bcb887f55cb3702b0e P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20251023 02:05:37.487923 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "816ef481cee84b259bc409b45a81baf1" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:37.389500 24831 raft_consensus.cc:515] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.450824 25005 raft_consensus.cc:2804] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader election won for term 8
W20251023 02:05:37.462508 25017 log.cc:927] Time spent T de76561746ce42549f51431c3f40bed7 P 818907d63f1f4d84ba267ebe6ee837e0: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
I20251023 02:05:37.517413 25022 raft_consensus.cc:2749] T 40b987b2b0ed4294b7ccf057ea33c0a0 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20251023 02:05:37.464015 24971 log.cc:927] Time spent T 1aaf7b9acded449f9017f47455026853 P ca1362b9d9b34073bf35b6f6c9d1159a: Append to log took a long time: real 0.855s user 0.000s sys 0.005s
I20251023 02:05:37.471397 24878 raft_consensus.cc:493] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
W20251023 02:05:37.466286 25001 log.cc:927] Time spent T cf578f7b0cdb4188bb43d676b1c64e28 P 02c53ba06cc3445a8534a84aa76b1c14: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20251023 02:05:37.450779 22619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325fa135bce480eaaa5faf74daf54f8" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" is_pre_election: true
I20251023 02:05:37.504361 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66c8e1c6015d47879ab7927548ad3b02" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:37.470943 24949 raft_consensus.cc:493] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:37.467166 25011 raft_consensus.cc:2804] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251023 02:05:37.445477 25024 consensus_queue.cc:1048] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.072s
I20251023 02:05:37.498525 24507 raft_consensus.cc:2749] T 90dcf9b1f8874437a7cd406b2ace7eeb P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251023 02:05:37.517553 21958 raft_consensus.cc:2468] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ab8b0b58147446d84d901b4ed2ff39e in term 10.
I20251023 02:05:37.518436 22036 raft_consensus.cc:2410] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20251023 02:05:37.539377 21962 raft_consensus.cc:1240] T d2b6e2b19ddc453290a76212731a2f28 P 02c53ba06cc3445a8534a84aa76b1c14 [term 10 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20251023 02:05:37.516917 22076 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.1:41499, user_credentials={real_user=slave}} blocked reactor thread for 155689us
I20251023 02:05:37.463222 25035 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
W20251023 02:05:37.470721 25023 log.cc:927] Time spent T fe5d794166cc4fbb87c6b918fa26b592 P 1ab8b0b58147446d84d901b4ed2ff39e: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20251023 02:05:37.492177 24346 raft_consensus.cc:493] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:37.498440 24870 raft_consensus.cc:515] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.448479 24968 consensus_queue.cc:260] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 7.2, Last appended by leader: 2, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.468937 24576 raft_consensus.cc:515] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.561321 22076 leader_election.cc:304] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters: 1ab8b0b58147446d84d901b4ed2ff39e
I20251023 02:05:37.548342 21961 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53734 (request call id 1694) took 2091 ms. Trace:
I20251023 02:05:37.566062 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24a8a5ca5c0e4c6384f51352dd78f5d0" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:37.479628 24924 raft_consensus.cc:515] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.549180 24850 consensus_queue.cc:1048] T 1376a6af91b24e46859bf630618fcec3 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251023 02:05:37.545890 24798 raft_consensus.cc:493] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1ab8b0b58147446d84d901b4ed2ff39e)
I20251023 02:05:37.554352 25024 raft_consensus.cc:493] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:37.554538 24740 leader_election.cc:290] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 02c53ba06cc3445a8534a84aa76b1c14 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:37.554618 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774c010489aa4775bb6d636926900529" candidate_uuid: "818907d63f1f4d84ba267ebe6ee837e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a"
I20251023 02:05:37.542516 25035 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:37.527827 24878 raft_consensus.cc:3060] T 972d07041fbd4097854ef23c73f09c75 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:37.524926 25005 raft_consensus.cc:697] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:37.560051 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd7f335f5df946c481e05b1a2a95f182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:37.565335 24507 raft_consensus.cc:2749] T f932b6202f5b497e8fb358af3c5d3f66 P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251023 02:05:37.521020 25003 raft_consensus.cc:697] T 6dddaeefc3d449a0ba9b3e6695e37507 P 818907d63f1f4d84ba267ebe6ee837e0 [term 10 LEADER]: Becoming Leader. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:37.566637 22036 raft_consensus.cc:3060] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:37.601740 22106 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56894 (request call id 303) took 1745 ms. Trace:
I20251023 02:05:37.538481 24949 raft_consensus.cc:3060] T c0a3abec901e4c2eba45233d075d9263 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:37.538601 25011 raft_consensus.cc:493] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:37.517560 25033 raft_consensus.cc:2804] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:37.566278 21961 rpcz_store.cc:276] 1023 02:05:35.457060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:35.457212 (+ 152us) service_pool.cc:225] Handling call
1023 02:05:35.457596 (+ 384us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:36.393390 (+935794us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:37.548111 (+1154721us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b540017c854. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:37.548271 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2150963968}
I20251023 02:05:37.534456 25038 raft_consensus.cc:2804] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader election won for term 7
I20251023 02:05:37.587679 22621 raft_consensus.cc:1240] T 365d47bf062b4ebabb6d5f247e4e3e33 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Rejecting Update request from peer ca1362b9d9b34073bf35b6f6c9d1159a for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20251023 02:05:37.610738 22029 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:37.556973 24346 raft_consensus.cc:3060] T 7ba6e9fe4bb44e559397893fa9d899db P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:37.595650 25035 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:37.597458 22103 raft_consensus.cc:1240] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [term 11 FOLLOWER]: Rejecting Update request from peer 818907d63f1f4d84ba267ebe6ee837e0 for earlier term 10. Current term is 11. Ops: []
I20251023 02:05:37.565409 22104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39662 (request call id 1755) took 1970 ms. Trace:
I20251023 02:05:37.615389 22104 rpcz_store.cc:276] 1023 02:05:35.595391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:35.597749 (+ 2358us) service_pool.cc:225] Handling call
1023 02:05:35.598106 (+ 357us) raft_consensus.cc:1471] Updating replica for 0 ops
1023 02:05:36.347408 (+749302us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:37.565171 (+1217763us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b54002fadd4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa999a1 00007f9c5fa983de 00007f9c64a419d9 00007f9c5dfd6d34 00007f9c5dfd6cc2 00007f9c5dfd6bf1 00007f9c5dfd6b6c 00007f9c5dfd5e22 00007f9c5ddd4ecc 00007f9c5ddd4306
1023 02:05:37.565347 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1698524416}
I20251023 02:05:37.583436 22619 raft_consensus.cc:2393] T 0325fa135bce480eaaa5faf74daf54f8 P 1ab8b0b58147446d84d901b4ed2ff39e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca1362b9d9b34073bf35b6f6c9d1159a in current term 8: Already voted for candidate 1ab8b0b58147446d84d901b4ed2ff39e in this term.
I20251023 02:05:37.599287 22076 leader_election.cc:304] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c53ba06cc3445a8534a84aa76b1c14, ca1362b9d9b34073bf35b6f6c9d1159a; no voters:
I20251023 02:05:37.602133 22106 rpcz_store.cc:276] 1023 02:05:35.856412 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:35.862635 (+ 6223us) service_pool.cc:225] Handling call
1023 02:05:35.862972 (+ 337us) raft_consensus.cc:1471] Updating replica for 1 ops
1023 02:05:35.863345 (+ 373us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 02:05:35.863370 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1023 02:05:35.863838 (+ 468us) log.cc:844] Serialized 26 byte log entry
1023 02:05:36.213531 (+349693us) raft_consensus.cc:1645] Marking committed up to 6
1023 02:05:36.213573 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 02:05:36.214283 (+ 710us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 02:05:37.463480 (+1249197us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b440002304c. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5c82bdc1 00007f9c5c8fa345 00007f9c5c8fad62 00007f9c5f641231 00007f9c5f63cb5a 00007f9c5f63091e 00007f9c5f62fc66 00007f9c5f62f888 00007f9c5f6392c1 00007f9c5f639279 00007f9c5f639209
1023 02:05:37.598642 (+135162us) raft_consensus.cc:1685] finished
1023 02:05:37.598662 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 02:05:37.601677 (+ 3015us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":47935,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":95,"spinlock_wait_cycles":32058624,"thread_start_us":349331,"threads_started":1,"wal-append.queue_time_us":668723}
I20251023 02:05:37.607476 22102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ed2c541deb4c8eb9997f54a56f68d7" candidate_uuid: "02c53ba06cc3445a8534a84aa76b1c14" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" is_pre_election: true
I20251023 02:05:37.617935 22102 raft_consensus.cc:2468] T d2ed2c541deb4c8eb9997f54a56f68d7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c53ba06cc3445a8534a84aa76b1c14 in term 7.
I20251023 02:05:37.608970 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57e843357de442f3a618eac9934513ed" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:37.596019 22107 raft_consensus.cc:2393] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 818907d63f1f4d84ba267ebe6ee837e0 in current term 10: Already voted for candidate ca1362b9d9b34073bf35b6f6c9d1159a in this term.
I20251023 02:05:37.557844 25022 consensus_queue.cc:1048] T c4c983d0648c4b649e08012b7932a15a P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.046s
I20251023 02:05:37.599093 24968 raft_consensus.cc:3060] T f09d991e8bee44c98b35a4bd632685bf P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Advancing to term 8
I20251023 02:05:37.623136 25005 consensus_queue.cc:237] T a48cb01caedc4f359126f3d85087d709 P 818907d63f1f4d84ba267ebe6ee837e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.623816 24875 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.615796 25038 raft_consensus.cc:697] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Becoming Leader. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:37.631138 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de76561746ce42549f51431c3f40bed7" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:37.597007 24798 raft_consensus.cc:515] T 816ef481cee84b259bc409b45a81baf1 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:37.609091 25011 raft_consensus.cc:3060] T 7600022898784926af7a5289ce03bc3a P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:37.634418 24924 leader_election.cc:290] T 5470b0fad25247c19157cd390de47f7f P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 11 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:37.674413 25042 raft_consensus.cc:493] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 02:05:37.675040 25042 raft_consensus.cc:515] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.675556 24928 raft_consensus.cc:3055] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 LEADER]: Stepping down as leader of term 9
I20251023 02:05:37.675937 24928 raft_consensus.cc:740] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 818907d63f1f4d84ba267ebe6ee837e0, State: Running, Role: LEADER
I20251023 02:05:37.676724 24928 consensus_queue.cc:260] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:37.677196 25042 leader_election.cc:290] T 74071b399ecc41808f0ac84dc836af4c P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:37.606374 24972 consensus_queue.cc:1059] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20251023 02:05:37.675031 24924 raft_consensus.cc:493] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 818907d63f1f4d84ba267ebe6ee837e0)
I20251023 02:05:37.678190 24924 raft_consensus.cc:515] T 98f1206f4e204b72af762bd083d766fd P ca1362b9d9b34073bf35b6f6c9d1159a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.610102 24292 raft_consensus.cc:2749] T 6dddaeefc3d449a0ba9b3e6695e37507 P ca1362b9d9b34073bf35b6f6c9d1159a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251023 02:05:37.610388 24610 raft_consensus.cc:2804] T 0325fa135bce480eaaa5faf74daf54f8 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader election won for term 8
I20251023 02:05:37.609004 25033 raft_consensus.cc:493] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251023 02:05:37.640607 22032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39012 (request call id 1357) took 1628 ms (client timeout 1536 ms). Trace:
W20251023 02:05:37.688449 22032 rpcz_store.cc:269] 1023 02:05:36.012418 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 02:05:36.073120 (+ 60702us) service_pool.cc:225] Handling call
1023 02:05:37.638216 (+1565096us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b540020c1d4. stack: 00007f9c5c9b9463 00007f9c5c9b9360 00007f9c5c30c574 00007f9c65fb5b14 00007f9c65fb5aa9 00007f9c5fa024c1 00007f9c5fa9ce8a 00007f9c64a430d7 00007f9c5dfdb834 00007f9c5dfdb7c2 00007f9c5dfdb6f1 00007f9c5dfdb66c 00007f9c5dfda922 00007f9c5ddd4ecc 00007f9c5ddd4306 00007f9c5ddd3caf
1023 02:05:37.640538 (+ 2322us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1852245248}
I20251023 02:05:37.642796 22035 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0594e36e0824e13965498e412eee3fb" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
I20251023 02:05:37.634322 24870 leader_election.cc:290] T d80086b174ba4cee80e75f95401994e3 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 6 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:37.642401 24558 raft_consensus.cc:2804] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251023 02:05:37.647131 24831 leader_election.cc:290] T 40b987b2b0ed4294b7ccf057ea33c0a0 P 818907d63f1f4d84ba267ebe6ee837e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), ca1362b9d9b34073bf35b6f6c9d1159a (127.20.81.3:33899)
I20251023 02:05:37.647670 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770eed9a6198463c889a1b9a978bd182" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:37.635310 24792 raft_consensus.cc:515] T ffe0e348fc464e36be33082ddcdba159 P 1ab8b0b58147446d84d901b4ed2ff39e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:37.649701 25029 raft_consensus.cc:3055] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 LEADER]: Stepping down as leader of term 8
I20251023 02:05:37.649313 25026 raft_consensus.cc:2955] T 5e923b66ccf04dd8985bbfe31adbf0b4 P 1ab8b0b58147446d84d901b4ed2ff39e [term 5 LEADER]: Committing config change with OpId 5.7: config changed from index 3 to 7, VOTER 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } } }
I20251023 02:05:37.658602 25038 consensus_queue.cc:237] T 0dc4e5ba4d0d4809a19f394786d00288 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
W20251023 02:05:37.662886 22003 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251023 02:05:37.607693 25036 consensus_queue.cc:1059] T cf578f7b0cdb4188bb43d676b1c64e28 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.037s
I20251023 02:05:37.666896 24576 leader_election.cc:290] T f18823a673d148e892e11c43e6430242 P ca1362b9d9b34073bf35b6f6c9d1159a [CANDIDATE]: Term 10 election: Requested vote from peers 02c53ba06cc3445a8534a84aa76b1c14 (127.20.81.1:41499), 818907d63f1f4d84ba267ebe6ee837e0 (127.20.81.2:40645)
I20251023 02:05:37.594864 25024 raft_consensus.cc:515] T 864751dbbbb748769206d89ccd8cb70e P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } attrs { replace: true } } peers { permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false } }
I20251023 02:05:37.680557 22034 raft_consensus.cc:2468] T 6869bc9cdea241aaa6d86d46a1b5bcfb P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:37.684381 22036 raft_consensus.cc:2468] T 24a8a5ca5c0e4c6384f51352dd78f5d0 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9.
I20251023 02:05:37.690106 22035 raft_consensus.cc:2468] T c0594e36e0824e13965498e412eee3fb P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 8.
I20251023 02:05:37.707500 22036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb1e5f57f70948bcb887f55cb3702b0e" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:37.708158 22036 raft_consensus.cc:2376] T fb1e5f57f70948bcb887f55cb3702b0e P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 9. Re-sending same reply.
I20251023 02:05:37.694630 22030 raft_consensus.cc:1240] T c954cadc886d47b39ed2cf51f9ac59f7 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Rejecting Update request from peer 02c53ba06cc3445a8534a84aa76b1c14 for earlier term 7. Current term is 8. Ops: []
I20251023 02:05:37.678457 24972 raft_consensus.cc:3055] T a48cb01caedc4f359126f3d85087d709 P ca1362b9d9b34073bf35b6f6c9d1159a [term 7 LEADER]: Stepping down as leader of term 7
I20251023 02:05:37.615353 24883 consensus_queue.cc:1059] T 365d47bf062b4ebabb6d5f247e4e3e33 P ca1362b9d9b34073bf35b6f6c9d1159a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ab8b0b58147446d84d901b4ed2ff39e" member_type: VOTER last_known_addr { host: "127.20.81.4" port: 33255 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.072s
I20251023 02:05:37.691267 22037 raft_consensus.cc:2376] T 770eed9a6198463c889a1b9a978bd182 P 818907d63f1f4d84ba267ebe6ee837e0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ca1362b9d9b34073bf35b6f6c9d1159a in term 7. Re-sending same reply.
I20251023 02:05:37.716069 22037 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5470b0fad25247c19157cd390de47f7f" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0" is_pre_election: true
W20251023 02:05:37.711335 22077 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.81.2:40645, user_credentials={real_user=slave}} blocked reactor thread for 289225us
I20251023 02:05:37.696313 25029 raft_consensus.cc:740] T d3bb66d66a264e8abdf220531328f12b P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ca1362b9d9b34073bf35b6f6c9d1159a, State: Running, Role: LEADER
I20251023 02:05:37.713203 24928 raft_consensus.cc:3060] T d2b6e2b19ddc453290a76212731a2f28 P 818907d63f1f4d84ba267ebe6ee837e0 [term 9 FOLLOWER]: Advancing to term 10
I20251023 02:05:37.689863 25033 raft_consensus.cc:3060] T edc3c9b5058946bf8f0da8fe69900291 P 818907d63f1f4d84ba267ebe6ee837e0 [term 8 FOLLOWER]: Advancing to term 9
I20251023 02:05:37.643425 24892 raft_consensus.cc:515] T c954cadc886d47b39ed2cf51f9ac59f7 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c53ba06cc3445a8534a84aa76b1c14" member_type: VOTER last_known_addr { host: "127.20.81.1" port: 41499 } } peers { permanent_uuid: "818907d63f1f4d84ba267ebe6ee837e0" member_type: VOTER last_known_addr { host: "127.20.81.2" port: 40645 } } peers { permanent_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" member_type: VOTER last_known_addr { host: "127.20.81.3" port: 33899 } }
I20251023 02:05:37.693646 24558 raft_consensus.cc:493] T 6ead1daa70a847959916a302d8540676 P ca1362b9d9b34073bf35b6f6c9d1159a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 02:05:37.710204 22032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f09d991e8bee44c98b35a4bd632685bf" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"
I20251023 02:05:37.711827 22034 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edc3c9b5058946bf8f0da8fe69900291" candidate_uuid: "ca1362b9d9b34073bf35b6f6c9d1159a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "818907d63f1f4d84ba267ebe6ee837e0"